Nov 01 22:22:56.137008 ip-10-0-130-144 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 22:22:56.550149 ip-10-0-130-144 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:56.550149 ip-10-0-130-144 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 22:22:56.550149 ip-10-0-130-144 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:56.550149 ip-10-0-130-144 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:22:56.555378 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555305    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.555378 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555373    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.555378 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555379    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.555378 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555383    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555387    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555392    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555396    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555401    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555405    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555408    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555412    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555417    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555421    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555425    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555428    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555432    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555439    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555445    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555452    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555456    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555460    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555464    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.555614 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555468    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555473    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555477    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555481    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555484    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555489    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555493    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555497    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555501    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555506    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555510    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555514    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555518    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555522    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555527    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555532    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555536    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555540    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555547    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.556420 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555554    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555559    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555565    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555569    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555573    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555589    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555594    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555599    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555603    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555607    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555612    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555617    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555621    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555626    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555630    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555634    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555638    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555643    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555647    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555652    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.557200 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555656    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555660    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555665    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555669    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555673    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555677    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555681    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555686    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555692    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555696    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555701    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555705    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555709    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555713    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555718    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555722    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555742    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555746    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555750    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.558047 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555754    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555758    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555762    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555767    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555772    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555776    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555781    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555785    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555789    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555793    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555797    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555802    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555806    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555810    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555814    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555819    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555823    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555827    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555833    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555837    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.558873 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555842    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555847    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555851    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555855    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555860    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.555864    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557130    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557140    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557145    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557149    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557154    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557158    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557162    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557167    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557171    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557176    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557181    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557186    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557190    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.559475 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557194    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557198    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557202    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557206    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557210    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557214    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557218    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557222    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557226    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557230    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557235    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557239    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557244    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557248    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557253    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557258    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557262    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557266    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557271    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.559957 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557275    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557279    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557284    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557288    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557292    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557297    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557301    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557305    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557310    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557314    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557318    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557322    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557327    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557332    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557336    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557340    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557344    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557349    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557353    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557358    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.560441 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557362    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557366    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557370    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557375    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557379    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557383    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557388    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557392    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557400    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557407    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557412    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557417    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557421    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557426    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557430    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557434    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557438    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557442    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557446    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.561326 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557451    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557455    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557459    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557463    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557467    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557472    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557477    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557481    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557485    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557489    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557494    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557499    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557503    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557510    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557516    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557521    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557526    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557531    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557536    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.562152 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557541    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557546    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557550    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557554    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557558    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557563    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557567    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557571    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557576    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557580    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557584    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557588    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557593    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557600    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557603    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.557608    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558437    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558445    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558450    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558456    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.562850 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558460    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558464    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558469    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558474    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558478    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558482    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558487    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558491    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558495    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558499    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558505    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558512    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558518    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558522    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558527    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558531    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558535    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558539    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558543    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.563323 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558548    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558552    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558556    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558560    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558565    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558570    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558574    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558587    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558592    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558596    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558601    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558605    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558610    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558614    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558619    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558623    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558627    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558631    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558635    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.564014 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558639    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558644    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558648    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558652    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558656    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558660    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558665    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558670    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558674    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558678    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558682    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558691    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558698    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558704    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558708    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558713    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558717    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558721    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558740    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558745    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.564462 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558748    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558756    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558760    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558765    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558769    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558774    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558778    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558783    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558787    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558792    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558796    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558801    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558805    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558809    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558814    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558818    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558822    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558826    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558831    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558835    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.565021 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558840    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558844    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558848    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558852    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558856    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558861    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558865    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558869    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558874    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558878    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558881    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558885    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558889    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558896    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558900    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558905    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558909    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558913    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558918    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.565541 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558922    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558927    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558931    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558937    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.558941    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560368    2579 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560385    2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560395    2579 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560402    2579 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560409    2579 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560415    2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560426    2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560432    2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560437    2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560442    2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560447    2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560452    2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560457    2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560461    2579 flags.go:64] FLAG: --cgroup-root=""
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560466    2579 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560471    2579 flags.go:64] FLAG: --client-ca-file=""
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560476    2579 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560481    2579 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560487    2579 flags.go:64] FLAG: --cluster-domain=""
Nov 01 22:22:56.566062 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560492    2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560497    2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560502    2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560508    2579 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560514    2579 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560519    2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560524    2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560529    2579 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560537    2579 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560541    2579 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560547    2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560552    2579 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560557    2579 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560564    2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560570    2579 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560575    2579 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560581    2579 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560586    2579 flags.go:64] FLAG: --enable-server="true"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560590    2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560598    2579 flags.go:64] FLAG: --event-burst="100"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560603    2579 flags.go:64] FLAG: --event-qps="50"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560608    2579 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560613    2579 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560618    2579 flags.go:64] FLAG: --eviction-hard=""
Nov 01 22:22:56.566620 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560625    2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560629    2579 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560634    2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560639    2579 flags.go:64] FLAG: --eviction-soft=""
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560644    2579 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560649    2579 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560654    2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560659    2579 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560663    2579 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560668    2579 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560673    2579 flags.go:64] FLAG: --feature-gates=""
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560679    2579 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560684    2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560689    2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560695    2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560700    2579 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560706    2579 flags.go:64] FLAG: --help="false"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560711    2579 flags.go:64] FLAG: --hostname-override="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560716    2579 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560722    2579 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560744    2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560750    2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560756    2579 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 22:22:56.567218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560761    2579 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560766    2579 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560770    2579 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560775    2579 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560780    2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560785    2579 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560789    2579 flags.go:64] FLAG: --kube-reserved=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560795    2579 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560799    2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560804    2579 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560809    2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560813    2579 flags.go:64] FLAG: --lock-file=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560818    2579 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560822    2579 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560827    2579 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560837    2579 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560842    2579 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560846    2579 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560851    2579 flags.go:64] FLAG: --logging-format="text"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560856    2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560862    2579 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560866    2579 flags.go:64] FLAG: --manifest-url=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560871    2579 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560878    2579 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560883    2579 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 22:22:56.567804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560891    2579 flags.go:64] FLAG: --max-pods="110"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560896    2579 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560901    2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560905    2579 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560911    2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560916    2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560921    2579 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560926    2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560937    2579 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560942    2579 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560946    2579 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560952    2579 flags.go:64] FLAG: --pod-cidr=""
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560956    2579 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560961    2579 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560965    2579 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560970    2579 flags.go:64] FLAG: --port="10250"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560975    2579 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560980    2579 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08e12a394936d5a8a"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560985    2579 flags.go:64] FLAG: --qos-reserved=""
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560990    2579 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560995    2579 flags.go:64] FLAG: --register-node="true"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.560999    2579 flags.go:64] FLAG: --register-with-taints=""
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561005    2579 flags.go:64] FLAG: --registry-burst="10"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561010    2579 flags.go:64] FLAG: --registry-qps="5"
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561015    2579 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561020    2579 flags.go:64] FLAG: --reserved-memory=""
Nov 01 22:22:56.568610 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561026    2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561031    2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561036    2579 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561040    2579 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561045    2579 flags.go:64] FLAG: --runonce="false"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561050    2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561056    2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561061    2579 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561066    2579 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561070    2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561075    2579 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561081    2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561086    2579 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561091    2579 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561095    2579 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561100    2579 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561105    2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561110    2579 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561114    2579 flags.go:64] FLAG: --system-cgroups=""
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561119    2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561127    2579 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561132    2579 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561136    2579 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561143    2579 flags.go:64] FLAG: --tls-min-version=""
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561147    2579 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 22:22:56.569472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561152    2579 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561157    2579 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561161    2579 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561166    2579 flags.go:64] FLAG: --v="2"
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561173    2579 flags.go:64] FLAG: --version="false"
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561179    2579 flags.go:64] FLAG: --vmodule=""
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561185    2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561193    2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561406    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561412    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561418    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561423    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561428    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561434    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561439    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561444    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561449    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561453    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561457    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561462    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561466    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.570249 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561470    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561474    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561478    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561483    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561487    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561491    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561496    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561500    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561504    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561509    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561513    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561517    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561522    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561526    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561530    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561534    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561539    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561543    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561549    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561553    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.570771 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561557    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561561    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561565    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561570    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561575    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561579    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561584    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561588    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561593    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561597    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561601    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561606    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561610    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561615    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561620    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561624    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561631    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561638    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561642    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561647    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.571253 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561651    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561655    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561660    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561664    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561668    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561672    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561676    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561680    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561684    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561688    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561694    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561699    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561703    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561707    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561711    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561716    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561721    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561740    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561745    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561749    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.571788 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561754    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561758    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561762    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561767    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561772    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561777    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561781    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561785    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561790    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561794    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561798    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561802    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561806    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561811    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561815    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561819    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561824    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561828    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561832    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.572268 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561836    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561841    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561845    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561851    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561856    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561860    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561864    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561868    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561872    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561880    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561887    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561892    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561897    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.561902    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.561946    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562002    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.572809 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562008    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562013    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562018    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562023    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562027    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562031    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562035    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562039    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562043    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562048    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562052    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562056    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562061    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562065    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562069    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562073    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562077    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562082    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562087    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562093    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.573190 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562100    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562105    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562110    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562114    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562118    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562122    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562128    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562132    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562136    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562140    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562145    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562149    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562153    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562157    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562161    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562165    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562169    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562173    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562177    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562182    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.573650 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562186    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562190    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562194    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562199    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562203    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562207    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562211    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562215    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562219    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562224    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562228    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562234    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562239    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562243    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562247    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562251    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562256    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562260    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562264    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.574141 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562269    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562273    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562278    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562282    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562286    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562290    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562295    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562300    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562303    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562308    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562313    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562317    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562321    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562327    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562335    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562339    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562343    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562348    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562352    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.574660 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562357    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562362    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562366    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562370    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562374    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562380    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562384    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562389    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562393    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562397    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562402    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562406    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562410    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562415    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562419    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562423    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562427    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562431    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562435    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.575162 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562440    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562444    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562449    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562453    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562458    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562463    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562468    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.562472    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.570654    2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.570670    2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.570685    2579 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.570690    2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570760    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570766    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570769    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570772    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570775    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570778    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570781    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.575717 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570784    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570787    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570790    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570793    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570795    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570798    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570802    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570804    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570807    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570809    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570812    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570815    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570821    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570824    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570827    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570829    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570833    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570835    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570838    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.576192 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570841    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570845    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570848    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570852    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570855    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570858    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570861    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570863    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570866    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570869    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570872    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570874    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570877    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570880    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570882    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570885    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570887    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570890    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570893    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.576699 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570895    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570898    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570901    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570903    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570906    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570909    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570911    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570914    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570916    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570919    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570922    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570924    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570927    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570930    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570932    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570935    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570938    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570940    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570943    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570946    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.577216 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570948    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570951    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570953    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570956    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570958    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570961    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570964    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570966    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570968    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570971    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570974    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570977    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570980    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570982    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570985    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570988    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570990    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570993    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570995    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.570998    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.578046 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571000    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571003    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571006    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571008    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571010    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571013    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571016    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571018    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571021    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571023    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571026    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571028    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571032    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571036    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571039    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571042    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571045    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571048    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571050    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.578569 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571053    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571055    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.571082    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571223    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571229    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571232    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571235    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571238    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571242    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571247    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571250    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571253    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571256    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571259    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.579069 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571262    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571264    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571267    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571270    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571273    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571275    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571278    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571280    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571283    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571285    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571288    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571291    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571293    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571296    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571299    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571301    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571304    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571307    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571309    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571312    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.579619 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571315    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571317    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571320    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571322    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571325    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571328    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571331    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571334    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571337    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571340    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571342    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571345    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571347    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571350    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571352    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571355    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571358    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571360    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571363    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.580171 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571365    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571368    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571370    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571373    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571375    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571378    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571381    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571384    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571386    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571389    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571391    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571394    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571396    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571399    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571401    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571404    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571406    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571409    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571411    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571414    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.580621 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571416    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571420    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571422    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571425    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571427    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571430    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571432    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571435    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571437    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571440    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571442    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571445    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571447    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571450    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571452    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571455    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571457    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571460    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571462    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571465    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571467    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.581144 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571470    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571472    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571475    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571477    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571480    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571482    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571485    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571487    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571490    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571492    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571495    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571497    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571500    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571503    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571505    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.571528    2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:22:56.581649 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571558    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571563    2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571566    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571569    2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571574    2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571577    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571580    2579 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571583    2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571585    2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571588    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571591    2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571594    2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571596    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571599    2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571601    2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571603    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571606    2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571609    2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571612    2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571614    2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:22:56.582057 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571617    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571619    2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571622    2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571626    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571629    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571631    2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571634    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571637    2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571639    2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571642    2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571645    2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571648    2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571651    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571653    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571656    2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571659    2579 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571663    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571666    2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571669    2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:22:56.582509 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571671    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571674    2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571677    2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571679    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571682    2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571685    2579 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571687    2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571690    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571692    2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571695    2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571697    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571700    2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571702    2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571705    2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571707    2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571710    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571715    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571718    2579 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571720    2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571737    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:22:56.582998 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571741    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571745    2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571749    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571753    2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571758    2579 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571762    2579 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571766    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571768    2579 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571771    2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571775    2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571777    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571780    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571783    2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571785    2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571788    2579 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571791    2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571794    2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571797    2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571799    2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571802    2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571805    2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:22:56.583522 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571808    2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571810    2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571813    2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571815    2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571818    2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571820    2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571823    2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571827    2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571831    2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571833    2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571836    2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571838    2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571841    2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571843    2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571846    2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571850    2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571858    2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571861    2579 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571863    2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:22:56.584056 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571866    2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571871    2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571876    2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571880    2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571885    2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571888    2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:56.571891    2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.571945    2579 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.574343    2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.575311    2579 server.go:1014] "Starting client certificate rotation"
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.575418    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.575474    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.577465    2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:22:56.584625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.579475    2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:22:56.591704 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.591691    2579 log.go:25] "Validated CRI v1 runtime API"
Nov 01 22:22:56.595402 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.595388    2579 log.go:25] "Validated CRI v1 image API"
Nov 01 22:22:56.596140 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.596116    2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 22:22:56.600452 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.600426    2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 910a2775-8e62-465c-b996-6049289d113e:/dev/nvme0n1p3 ec8b3902-7669-4b82-a531-50a3ef46780e:/dev/nvme0n1p4]
Nov 01 22:22:56.600505 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.600452    2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 22:22:56.607102 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.606990    2579 manager.go:217] Machine: {Timestamp:2026-11-01 22:22:56.605747147 +0000 UTC m=+0.354152840 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3109664 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26f4f7aed77ba0380b8f48a6c0615b SystemUUID:ec26f4f7-aed7-7ba0-380b-8f48a6c0615b BootID:3fddff22-5a75-4577-af8b-47b9b183ac1f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:ef:ca:6c:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:ef:ca:6c:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:6e:96:1c:c3:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 22:22:56.607102 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.607094    2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 22:22:56.607238 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.607185    2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 22:22:56.610691 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.610663    2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 22:22:56.610829 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.610693    2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-144.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 22:22:56.610880 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.610832    2579 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 22:22:56.610880 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.610840    2579 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 22:22:56.610880 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.610862    2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:22:56.611847 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.611837    2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:22:56.613052 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.613043    2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 22:22:56.617433 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.617422    2579 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 22:22:56.617545 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.617535    2579 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 22:22:56.618369 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.618359    2579 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 22:22:56.618415 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.618376    2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 22:22:56.618415 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.618392    2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 22:22:56.618415 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.618403    2579 kubelet.go:405] "Adding apiserver pod source"
Nov 01 22:22:56.618415 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.618413    2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 22:22:56.618715 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.618699    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:22:56.621159 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.621143    2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 22:22:56.627919 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.627906    2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:22:56.628936 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.628924    2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 22:22:56.628979 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.628954    2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 22:22:56.629118 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629104    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 22:22:56.629118 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629120    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 22:22:56.629234 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629127    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 22:22:56.629234 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629132    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 22:22:56.629234 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629138    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 22:22:56.629234 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629145    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 22:22:56.629907 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629890    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 22:22:56.629907 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629914    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 22:22:56.629975 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629922    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 22:22:56.629975 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629928    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 22:22:56.629975 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629941    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 22:22:56.629975 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.629952    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 22:22:56.631800 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.631784    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 22:22:56.631800 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.631795    2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 22:22:56.635375 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.635362    2579 server.go:1263] "Started kubelet"
Nov 01 22:22:56.635948 ip-10-0-130-144 systemd[1]: Started Kubernetes Kubelet.
Nov 01 22:22:56.637755 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.637686    2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 22:22:56.637812 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.637776    2579 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 22:22:56.638690 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.638656    2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 22:22:56.639997 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.639981    2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 22:22:56.640588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.640577    2579 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 22:22:56.642957 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.642937    2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 22:22:56.645494 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.645475    2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 22:22:56.646755 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.646740    2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 22:22:56.646878 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.646853    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.647577 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647557    2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 22:22:56.647577 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647558    2579 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 22:22:56.647708 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647586    2579 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 22:22:56.647708 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647605    2579 factory.go:55] Registering systemd factory
Nov 01 22:22:56.647708 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647616    2579 factory.go:223] Registration of the systemd container factory successfully
Nov 01 22:22:56.647826 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647713    2579 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 22:22:56.647826 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647723    2579 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 22:22:56.647916 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647842    2579 factory.go:153] Registering CRI-O factory
Nov 01 22:22:56.647916 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647858    2579 factory.go:223] Registration of the crio container factory successfully
Nov 01 22:22:56.647916 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647904    2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 22:22:56.648018 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647921    2579 factory.go:103] Registering Raw factory
Nov 01 22:22:56.648018 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.647936    2579 manager.go:1199] Started watching for new ooms in manager
Nov 01 22:22:56.649027 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.649001    2579 manager.go:319] Starting recovery of all containers
Nov 01 22:22:56.658593 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.658567    2579 manager.go:324] Recovery completed
Nov 01 22:22:56.662509 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.662496    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.664185 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664163    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.664263 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664196    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.664263 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664206    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.664740 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664717    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.664740 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664720    2579 cpu_manager.go:226] "Starting" policy="none"
Nov 01 22:22:56.664877 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664748    2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 22:22:56.664877 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.664767    2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 22:22:56.665358 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.665345    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.665419 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.665368    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.665419 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.665378    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.667059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.667046    2579 policy_none.go:50] "Start"
Nov 01 22:22:56.667129 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.667064    2579 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 22:22:56.667129 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.667076    2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 22:22:56.669136 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.669124    2579 policy_none.go:44] "Start"
Nov 01 22:22:56.691338 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.691307    2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 22:22:56.695106 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695090    2579 manager.go:341] "Starting Device Plugin manager"
Nov 01 22:22:56.695195 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.695116    2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 22:22:56.695195 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695124    2579 server.go:87] "Starting device plugin registration server"
Nov 01 22:22:56.695357 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695341    2579 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 22:22:56.695400 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695360    2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 22:22:56.695466 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695451    2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 22:22:56.695543 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695532    2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 22:22:56.695543 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.695541    2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 22:22:56.696192 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.696169    2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 22:22:56.696288 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.696227    2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:56.724992 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.717550    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 22:22:56.724992 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.718981    2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 22:22:56.724992 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.719009    2579 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 22:22:56.724992 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.719029    2579 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 22:22:56.724992 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.719066    2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 22:22:56.768892 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.768864    2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-144.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 22:22:56.768892 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.768872    2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-144.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 22:22:56.770002 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.768872    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-144.us-west-2.compute.internal.18e40c073183f634  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-144.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-144.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:56.635336244 +0000 UTC m=+0.383741936,LastTimestamp:2026-11-01 22:22:56.635336244 +0000 UTC m=+0.383741936,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-144.us-west-2.compute.internal,}"
Nov 01 22:22:56.796034 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.796014    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.796766 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.796752    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.796829 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.796779    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.796829 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.796790    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.796829 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.796809    2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.820470 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.820404    2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"]
Nov 01 22:22:56.820470 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.820459    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.821164 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.821148    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.821246 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.821177    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.821246 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.821191    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.822989 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.822975    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.823204 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823182    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.823249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823235    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.823631 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823608    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.823631 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823634    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.823798 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823643    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.823955 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823942    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.824008 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823966    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.824008 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.823975    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.824008 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.823948    2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-144.us-west-2.compute.internal.18e40c07333c2335  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-144.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-144.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-144.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:56.664183605 +0000 UTC m=+0.412589297,LastTimestamp:2026-11-01 22:22:56.664183605 +0000 UTC m=+0.412589297,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-144.us-west-2.compute.internal,}"
Nov 01 22:22:56.824412 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.824389    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Nov 01 22:22:56.824462 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.824393    2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-144.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 22:22:56.825486 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.825470    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.825553 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.825503    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:56.826122 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.826104    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:56.826194 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.826142    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:56.826194 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.826157    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:56.849282 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.849264    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 22:22:56.849360 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.849311    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6rzfh"
Nov 01 22:22:56.859474 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.859458    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.861234 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:56.861218    2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.915201 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.915181    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:56.949520 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.949500    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ad307b3c1f78fefabe49820df30682a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"6ad307b3c1f78fefabe49820df30682a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.949588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.949523    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:56.949588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:56.949539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.010153 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.010134    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:57.010769 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.010755    2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.010811 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.010773    2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-144.us-west-2.compute.internal\": node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.049648 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.049630    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.049709 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.049653    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.049709 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.049664    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.049709 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.049699    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.049826 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.049691    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ad307b3c1f78fefabe49820df30682a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"6ad307b3c1f78fefabe49820df30682a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.049826 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.049715    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ad307b3c1f78fefabe49820df30682a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"6ad307b3c1f78fefabe49820df30682a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.125333 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.125288    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.163762 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.163739    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.165862 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.165848    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:57.226073 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.225935    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.326930 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.326901    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.427866 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.427822    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.528619 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.528587    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.574982 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.574957    2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 22:22:57.575372 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.575074    2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a40c4aa6d7453489f8ffaf27c6ab6881-5876fe58ba2f5954.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-130-144.us-west-2.compute.internal.18e40c07333c2335\": read tcp 10.0.130.144:59798->18.219.135.16:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-130-144.us-west-2.compute.internal.18e40c07333c2335  default   5042 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-144.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-144.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-144.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 22:22:56 +0000 UTC,LastTimestamp:2026-11-01 22:22:56.821163015 +0000 UTC m=+0.569568711,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-144.us-west-2.compute.internal,}"
Nov 01 22:22:57.575372 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.575138    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 22:22:57.575372 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.575144    2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 22:22:57.629411 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.629390    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.729781 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.729722    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.830411 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.830392    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.854650 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.854625    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-07-27 22:38:07.176960766 +0000 UTC"
Nov 01 22:22:57.854650 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.854646    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15216h15m9.322317944s"
Nov 01 22:22:57.888974 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.888952    2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:22:57.889779 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.889765    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:22:57.889860 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.889796    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:22:57.889860 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.889810    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:22:57.907365 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.907346    2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:57.931423 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:57.931407    2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:22:57.954878 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:57.954858    2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:58.046990 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.046949    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:58.112937 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.112921    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:22:58.113753 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.113742    2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:22:58.171850 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.171835    2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:22:58.308760 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:58.308694    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6403352f0c73b775f9816057f7f4c51.slice/crio-a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630 WatchSource:0}: Error finding container a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630: Status 404 returned error can't find the container with id a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630
Nov 01 22:22:58.315218 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.315203    2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:22:58.383984 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:58.383955    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad307b3c1f78fefabe49820df30682a.slice/crio-05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a WatchSource:0}: Error finding container 05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a: Status 404 returned error can't find the container with id 05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a
Nov 01 22:22:58.620034 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.619985    2579 apiserver.go:52] "Watching apiserver"
Nov 01 22:22:58.647545 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.647526    2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 22:22:58.672633 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.672612    2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:58.723259 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.723220    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"6ad307b3c1f78fefabe49820df30682a","Type":"ContainerStarted","Data":"05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a"}
Nov 01 22:22:58.724313 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.724290    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"b6403352f0c73b775f9816057f7f4c51","Type":"ContainerStarted","Data":"a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630"}
Nov 01 22:22:58.775622 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.775600    2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:22:58.775700 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.775645    2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 22:22:58.775954 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.775937    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bn45z","kube-system/konnectivity-agent-ggq5x","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw","openshift-network-diagnostics/network-check-target-gw5f9","kube-system/global-pull-secret-syncer-v4lzz","openshift-cluster-node-tuning-operator/tuned-wdllg","openshift-image-registry/node-ca-nq2kt","openshift-multus/multus-h8rr2","openshift-multus/network-metrics-daemon-r9kzv","openshift-network-operator/iptables-alerter-nzcrt","kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal","openshift-dns/node-resolver-6hpf6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal","openshift-multus/multus-additional-cni-plugins-q9hwj"]
Nov 01 22:22:58.779514 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.779502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.782004 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.781969    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.784143 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.784128    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.786274 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.786258    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:22:58.786367 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.786344    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:22:58.788356 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.788337    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.790455 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.790439    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.792757 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.792742    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.794798 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.794784    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.797245 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.797228    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.797315 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.797300    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:22:58.799376 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.799359    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:58.799438 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.799424    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:22:58.801492 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.801476    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.803644 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.803625    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.833366 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.833350    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.833366 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.833352    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 22:22:58.833493 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.833464    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.833567 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.833557    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67p2k\""
Nov 01 22:22:58.835605 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.835589    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 22:22:58.835605 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.835588    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jbqbs\""
Nov 01 22:22:58.835751 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.835610    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.835751 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.835629    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.835751 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.835679    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 22:22:58.837157 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.837145    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.837297 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.837281    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.837351 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.837292    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 22:22:58.837351 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.837344    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wj42\""
Nov 01 22:22:58.841717 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.841701    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.841802 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.841761    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.841802 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.841788    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4xsx\""
Nov 01 22:22:58.843778 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.843760    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 22:22:58.843845 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.843783    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm5kv\""
Nov 01 22:22:58.843845 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.843816    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 22:22:58.846049 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.846034    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 22:22:58.846049 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.846038    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.846177 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.846071    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.846255 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.846243    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftrlf\""
Nov 01 22:22:58.848522 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.848508    2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 22:22:58.861507 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861486    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vbmf\" (UniqueName: \"kubernetes.io/projected/0e10c383-d5b0-4782-9b86-85e113ae6e1c-kube-api-access-9vbmf\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.861579 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861509    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9441fd42-7d78-4065-8f63-462144f681a4-host-slash\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.861579 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861526    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-kubelet\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.861579 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861539    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.861579 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861555    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fksbz\" (UniqueName: \"kubernetes.io/projected/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-kube-api-access-fksbz\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.861579 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861570    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c493649-a226-4fdc-a95a-ba1a9de15539-agent-certs\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.861739 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861625    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c493649-a226-4fdc-a95a-ba1a9de15539-konnectivity-ca\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.861739 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861657    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-netd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.861739 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861716    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.861830 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-kubelet\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.861830 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861803    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-conf-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.861889 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861829    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:22:58.861889 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861844    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-lib-modules\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.861889 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861872    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-tmp\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.861972 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861892    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r9ps\" (UniqueName: \"kubernetes.io/projected/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-kube-api-access-5r9ps\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.861972 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861919    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-device-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.861972 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861938    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-modprobe-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862058 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861974    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862058 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.861998    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-ovn\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862058 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862036    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.862058 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862057    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-kubernetes\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862168 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862070    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-host\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862168 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862107    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:58.862168 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862142    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-systemd-units\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862168 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862165    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-bin\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862279 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cnibin\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.862279 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-run\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862279 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862233    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cni-binary-copy\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862279 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862258    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9hs\" (UniqueName: \"kubernetes.io/projected/9441fd42-7d78-4065-8f63-462144f681a4-kube-api-access-6v9hs\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.862386 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862293    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-hosts-file\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.862386 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862317    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c14af69e-6fab-438d-82c2-4b0523bfba4a-serviceca\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.862386 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862332    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cnibin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862386 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862361    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-socket-dir-parent\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862500 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862385    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-var-lib-kubelet\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862500 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862402    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-slash\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862500 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862424    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862500 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862468    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-multus\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862500 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862484    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-registration-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.862638 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862502    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-kubelet-config\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.862638 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862529    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-netns\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862638 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862550    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trbxq\" (UniqueName: \"kubernetes.io/projected/c14af69e-6fab-438d-82c2-4b0523bfba4a-kube-api-access-trbxq\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.862638 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862593    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-system-cni-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.862638 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862614    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.862796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862642    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c14af69e-6fab-438d-82c2-4b0523bfba4a-host\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.862796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862659    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-socket-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.862796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862688    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6df\" (UniqueName: \"kubernetes.io/projected/f9f3380f-6866-4ade-b5b8-17416d4bae89-kube-api-access-jt6df\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:58.862796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862715    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-var-lib-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862751    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.862796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862782    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-k8s-cni-cncf-io\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862798    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-multus-certs\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862833    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-sys\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862857    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862873    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-config\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862898    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-system-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862921    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-daemon-config\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862937    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-sys-fs\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.862965 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862960    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-systemd\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.863183 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862976    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9441fd42-7d78-4065-8f63-462144f681a4-iptables-alerter-script\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.863183 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.862994    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-tmp-dir\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.863183 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863012    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.863183 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863033    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-log-socket\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.863183 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863051    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.863183 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863081    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjljb\" (UniqueName: \"kubernetes.io/projected/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kube-api-access-kjljb\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.863370 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863186    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-node-log\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.863414 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863360    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzlm\" (UniqueName: \"kubernetes.io/projected/589a7de9-d92b-43e5-98eb-604e083bd543-kube-api-access-wxzlm\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.863459 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863416    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-etc-kubernetes\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.863459 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863449    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.863598 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863513    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysconfig\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863770    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbnvw\" (UniqueName: \"kubernetes.io/projected/66738f34-062c-492c-9930-7f5882b3073d-kube-api-access-qbnvw\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863818    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-systemd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863870    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589a7de9-d92b-43e5-98eb-604e083bd543-ovn-node-metrics-cert\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863927    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-script-lib\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.863967    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-bin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864005    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-etc-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864053    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-etc-tuned\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864084    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864113    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-netns\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864162    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-conf\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-dbus\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864224    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-env-overrides\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.865354 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.864251    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-hostroot\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.881238 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881195    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.881418 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881401    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ljglf\""
Nov 01 22:22:58.881472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881442    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 22:22:58.881472 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881446    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 22:22:58.881578 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881565    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.881699 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881686    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hb4ll\""
Nov 01 22:22:58.881786 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881705    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2xg9\""
Nov 01 22:22:58.881880 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881865    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 22:22:58.881934 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881868    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 22:22:58.881987 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.881964    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 22:22:58.882121 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.882108    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 22:22:58.882165 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.882149    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 22:22:58.882230 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.882217    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 22:22:58.884839 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.884823    2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 22:22:58.884892 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.884859    2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-49jvg"
Nov 01 22:22:58.965134 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965114    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-netns\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965136    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbxq\" (UniqueName: \"kubernetes.io/projected/c14af69e-6fab-438d-82c2-4b0523bfba4a-kube-api-access-trbxq\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-system-cni-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965167    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965181    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c14af69e-6fab-438d-82c2-4b0523bfba4a-host\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965194    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-socket-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965207    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6df\" (UniqueName: \"kubernetes.io/projected/f9f3380f-6866-4ade-b5b8-17416d4bae89-kube-api-access-jt6df\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:58.965222 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965222    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-var-lib-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965231    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-system-cni-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965252    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-netns\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965271    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965276    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-var-lib-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c14af69e-6fab-438d-82c2-4b0523bfba4a-host\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965315    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-k8s-cni-cncf-io\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965294    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-k8s-cni-cncf-io\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965354    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-multus-certs\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965378    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-sys\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965395    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965414    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-config\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965426    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965437    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-system-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-multus-certs\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-daemon-config\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965472    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965466    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-sys\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.965552 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965492    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-sys-fs\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965536    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-sys-fs\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965540    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-systemd\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965582    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9441fd42-7d78-4065-8f63-462144f681a4-iptables-alerter-script\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965585    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-systemd\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965547    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-system-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965650    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-tmp-dir\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965674    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965689    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-log-socket\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965704    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjljb\" (UniqueName: \"kubernetes.io/projected/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kube-api-access-kjljb\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.965749    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965757    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-node-log\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965771    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzlm\" (UniqueName: \"kubernetes.io/projected/589a7de9-d92b-43e5-98eb-604e083bd543-kube-api-access-wxzlm\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965777    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-log-socket\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.965804    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret podName:ec65095d-39d7-47d4-8951-b9676b711b1d nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.465780105 +0000 UTC m=+3.214185784 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret") pod "global-pull-secret-syncer-v4lzz" (UID: "ec65095d-39d7-47d4-8951-b9676b711b1d") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965820    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-etc-kubernetes\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966059 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965838    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965852    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysconfig\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965869    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnvw\" (UniqueName: \"kubernetes.io/projected/66738f34-062c-492c-9930-7f5882b3073d-kube-api-access-qbnvw\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965883    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-systemd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965897    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589a7de9-d92b-43e5-98eb-604e083bd543-ovn-node-metrics-cert\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965913    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-script-lib\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965927    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-bin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965931    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-node-log\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965941    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-etc-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965955    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-etc-tuned\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965969    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965970    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysconfig\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965982    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-netns\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.965988    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-etc-kubernetes\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966011    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-netns\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966019    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966042    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-conf\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966044    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-bin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.966777 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966058    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-dbus\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966073    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-env-overrides\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966084    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966091    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-etc-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966087    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-hostroot\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966177    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-conf\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966188    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-socket-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966225    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-systemd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966348    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966373    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-hostroot\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966382    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbmf\" (UniqueName: \"kubernetes.io/projected/0e10c383-d5b0-4782-9b86-85e113ae6e1c-kube-api-access-9vbmf\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966545    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9441fd42-7d78-4065-8f63-462144f681a4-host-slash\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966549    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966570    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-kubelet\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966438    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-dbus\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966599    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966622    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9441fd42-7d78-4065-8f63-462144f681a4-host-slash\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966639    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksbz\" (UniqueName: \"kubernetes.io/projected/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-kube-api-access-fksbz\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.967588 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966648    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-kubelet\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966670    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c493649-a226-4fdc-a95a-ba1a9de15539-agent-certs\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966698    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c493649-a226-4fdc-a95a-ba1a9de15539-konnectivity-ca\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966722    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-netd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966783    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-netd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966826    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966851    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-daemon-config\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966877    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-kubelet\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966906    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-kubelet\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966912    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-config\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966912    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9441fd42-7d78-4065-8f63-462144f681a4-iptables-alerter-script\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966924    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-conf-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966951    2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.966964    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-conf-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967027    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967052    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-lib-modules\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967071    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-tmp\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.968416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967091    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9ps\" (UniqueName: \"kubernetes.io/projected/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-kube-api-access-5r9ps\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967111    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-device-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967129    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-modprobe-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967149    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967170    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-ovn\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967200    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967226    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-kubernetes\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967246    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-host\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967277    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967304    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-systemd-units\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967326    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-bin\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967348    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cnibin\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967366    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-run\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967386    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cni-binary-copy\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967402    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-tmp-dir\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967413    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9hs\" (UniqueName: \"kubernetes.io/projected/9441fd42-7d78-4065-8f63-462144f681a4-kube-api-access-6v9hs\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967433    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-script-lib\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.969272 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967500    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-hosts-file\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967439    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-hosts-file\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967536    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c14af69e-6fab-438d-82c2-4b0523bfba4a-serviceca\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967559    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cnibin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967588    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-socket-dir-parent\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967608    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-var-lib-kubelet\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967612    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-env-overrides\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967683    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-host\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967684    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-lib-modules\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967717    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-var-lib-kubelet\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967721    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c493649-a226-4fdc-a95a-ba1a9de15539-konnectivity-ca\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.967805    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:58.967852    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.467837619 +0000 UTC m=+3.216243322 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967875    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967886    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-systemd-units\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967924    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-bin\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967956    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cnibin\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967994    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-run\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.970077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967997    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c14af69e-6fab-438d-82c2-4b0523bfba4a-serviceca\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968051    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cnibin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968093    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-socket-dir-parent\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968120    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-device-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.967411    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968163    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-modprobe-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968189    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-ovn\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968346    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-slash\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968367    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cni-binary-copy\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968374    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968402    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-multus\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968423    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-registration-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968436    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-kubernetes\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968449    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-kubelet-config\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968470    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968532    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-kubelet-config\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968540    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-multus\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:58.971020 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968534    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-slash\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.971853 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.968573    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-registration-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:58.971853 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.969293    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:58.971853 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.970440    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589a7de9-d92b-43e5-98eb-604e083bd543-ovn-node-metrics-cert\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:58.971853 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.970466    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-etc-tuned\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:58.971853 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.970573    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c493649-a226-4fdc-a95a-ba1a9de15539-agent-certs\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:58.971999 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:58.971939    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-tmp\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:59.024167 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.024145    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjljb\" (UniqueName: \"kubernetes.io/projected/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kube-api-access-kjljb\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:59.024696 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.024665    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbxq\" (UniqueName: \"kubernetes.io/projected/c14af69e-6fab-438d-82c2-4b0523bfba4a-kube-api-access-trbxq\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:59.025037 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.025020    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbmf\" (UniqueName: \"kubernetes.io/projected/0e10c383-d5b0-4782-9b86-85e113ae6e1c-kube-api-access-9vbmf\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:59.025433 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.025414    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksbz\" (UniqueName: \"kubernetes.io/projected/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-kube-api-access-fksbz\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:59.025573 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.025553    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnvw\" (UniqueName: \"kubernetes.io/projected/66738f34-062c-492c-9930-7f5882b3073d-kube-api-access-qbnvw\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:59.025738 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.025710    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:59.025801 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.025743    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:59.025801 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.025756    2579 projected.go:196] Error preparing data for projected volume kube-api-access-hzpx7 for pod openshift-network-diagnostics/network-check-target-gw5f9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.025884 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.025813    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7 podName:01c3555d-f354-45dc-9ef9-3932727a8196 nodeName:}" failed. No retries permitted until 2026-11-01 22:22:59.525798398 +0000 UTC m=+3.274204102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hzpx7" (UniqueName: "kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7") pod "network-check-target-gw5f9" (UID: "01c3555d-f354-45dc-9ef9-3932727a8196") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.026335 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.026315    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzlm\" (UniqueName: \"kubernetes.io/projected/589a7de9-d92b-43e5-98eb-604e083bd543-kube-api-access-wxzlm\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:59.026414 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.026345    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6df\" (UniqueName: \"kubernetes.io/projected/f9f3380f-6866-4ade-b5b8-17416d4bae89-kube-api-access-jt6df\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:59.026885 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.026871    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9hs\" (UniqueName: \"kubernetes.io/projected/9441fd42-7d78-4065-8f63-462144f681a4-kube-api-access-6v9hs\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:59.027721 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.027705    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9ps\" (UniqueName: \"kubernetes.io/projected/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-kube-api-access-5r9ps\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:59.091776 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.091758    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:22:59.096847 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.096836    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-h8rr2"
Nov 01 22:22:59.100907 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.100882    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc14af69e_6fab_438d_82c2_4b0523bfba4a.slice/crio-c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527 WatchSource:0}: Error finding container c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527: Status 404 returned error can't find the container with id c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527
Nov 01 22:22:59.104890 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.104873    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e10c383_d5b0_4782_9b86_85e113ae6e1c.slice/crio-2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263 WatchSource:0}: Error finding container 2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263: Status 404 returned error can't find the container with id 2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263
Nov 01 22:22:59.109229 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.109211    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:22:59.109298 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.109215    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:22:59.113956 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.113944    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:22:59.118661 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.118637    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc93a44d2_477d_4501_b7ac_f681ba8f0ae9.slice/crio-88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09 WatchSource:0}: Error finding container 88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09: Status 404 returned error can't find the container with id 88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09
Nov 01 22:22:59.119207 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.119189    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:22:59.122651 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.122565    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66738f34_062c_492c_9930_7f5882b3073d.slice/crio-82343b81037dfa2a294f5a40b5e660e9f47c7d09bb9fa897e164b70d8d4b711f WatchSource:0}: Error finding container 82343b81037dfa2a294f5a40b5e660e9f47c7d09bb9fa897e164b70d8d4b711f: Status 404 returned error can't find the container with id 82343b81037dfa2a294f5a40b5e660e9f47c7d09bb9fa897e164b70d8d4b711f
Nov 01 22:22:59.124405 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.124372    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:22:59.124708 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.124690    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c493649_a226_4fdc_a95a_ba1a9de15539.slice/crio-dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370 WatchSource:0}: Error finding container dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370: Status 404 returned error can't find the container with id dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370
Nov 01 22:22:59.128402 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.128383    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9441fd42_7d78_4065_8f63_462144f681a4.slice/crio-90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11 WatchSource:0}: Error finding container 90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11: Status 404 returned error can't find the container with id 90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11
Nov 01 22:22:59.133270 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.133251    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c668ab_f0d3_45ee_a5c0_6f37c2ff8400.slice/crio-32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f WatchSource:0}: Error finding container 32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f: Status 404 returned error can't find the container with id 32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f
Nov 01 22:22:59.152163 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.152143    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:22:59.158804 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.158787    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:22:59.160837 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.160818    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod589a7de9_d92b_43e5_98eb_604e083bd543.slice/crio-1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d WatchSource:0}: Error finding container 1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d: Status 404 returned error can't find the container with id 1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d
Nov 01 22:22:59.166330 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:22:59.166307    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a25c3b_f254_46fb_a0dd_90f4c5cb4b9c.slice/crio-aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad WatchSource:0}: Error finding container aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad: Status 404 returned error can't find the container with id aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad
Nov 01 22:22:59.472000 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.471929    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:22:59.472000 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.471988    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:22:59.472158 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.472134    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:59.472208 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.472184    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret podName:ec65095d-39d7-47d4-8951-b9676b711b1d nodeName:}" failed. No retries permitted until 2026-11-01 22:23:00.472167889 +0000 UTC m=+4.220573576 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret") pod "global-pull-secret-syncer-v4lzz" (UID: "ec65095d-39d7-47d4-8951-b9676b711b1d") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:22:59.472250 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.472245    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:59.472287 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.472276    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:00.472265573 +0000 UTC m=+4.220671258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:22:59.576955 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.575320    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:22:59.576955 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.575538    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:22:59.576955 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.575554    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:22:59.576955 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.575567    2579 projected.go:196] Error preparing data for projected volume kube-api-access-hzpx7 for pod openshift-network-diagnostics/network-check-target-gw5f9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.576955 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:22:59.575612    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7 podName:01c3555d-f354-45dc-9ef9-3932727a8196 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:00.575594745 +0000 UTC m=+4.324000424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzpx7" (UniqueName: "kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7") pod "network-check-target-gw5f9" (UID: "01c3555d-f354-45dc-9ef9-3932727a8196") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:22:59.727103 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.727034    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzcrt" event={"ID":"9441fd42-7d78-4065-8f63-462144f681a4","Type":"ContainerStarted","Data":"90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11"}
Nov 01 22:22:59.728148 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.728086    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wdllg" event={"ID":"66738f34-062c-492c-9930-7f5882b3073d","Type":"ContainerStarted","Data":"82343b81037dfa2a294f5a40b5e660e9f47c7d09bb9fa897e164b70d8d4b711f"}
Nov 01 22:22:59.729148 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.729127    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw" event={"ID":"c93a44d2-477d-4501-b7ac-f681ba8f0ae9","Type":"ContainerStarted","Data":"88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09"}
Nov 01 22:22:59.730351 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.730330    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h8rr2" event={"ID":"0e10c383-d5b0-4782-9b86-85e113ae6e1c","Type":"ContainerStarted","Data":"2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263"}
Nov 01 22:22:59.731388 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.731363    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6hpf6" event={"ID":"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400","Type":"ContainerStarted","Data":"32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f"}
Nov 01 22:22:59.732520 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.732496    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ggq5x" event={"ID":"1c493649-a226-4fdc-a95a-ba1a9de15539","Type":"ContainerStarted","Data":"dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370"}
Nov 01 22:22:59.733558 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.733525    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nq2kt" event={"ID":"c14af69e-6fab-438d-82c2-4b0523bfba4a","Type":"ContainerStarted","Data":"c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527"}
Nov 01 22:22:59.734654 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.734631    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerStarted","Data":"aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad"}
Nov 01 22:22:59.736130 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.736112    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d"}
Nov 01 22:22:59.888107 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.888057    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:58 +0000 UTC" deadline="2028-04-17 19:06:29.251979382 +0000 UTC"
Nov 01 22:22:59.888107 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:22:59.888099    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12788h43m29.363883262s"
Nov 01 22:23:00.483128 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.483090    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:00.483287 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.483183    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:00.483404 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.483387    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:00.483469 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.483450    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret podName:ec65095d-39d7-47d4-8951-b9676b711b1d nodeName:}" failed. No retries permitted until 2026-11-01 22:23:02.483432607 +0000 UTC m=+6.231838304 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret") pod "global-pull-secret-syncer-v4lzz" (UID: "ec65095d-39d7-47d4-8951-b9676b711b1d") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:00.483546 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.483532    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:00.483602 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.483577    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:02.483564493 +0000 UTC m=+6.231970188 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:00.584010 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.583976    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:00.584182 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.584122    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:00.584182 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.584139    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:00.584182 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.584153    2579 projected.go:196] Error preparing data for projected volume kube-api-access-hzpx7 for pod openshift-network-diagnostics/network-check-target-gw5f9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:00.584350 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.584205    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7 podName:01c3555d-f354-45dc-9ef9-3932727a8196 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:02.584187914 +0000 UTC m=+6.332593598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzpx7" (UniqueName: "kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7") pod "network-check-target-gw5f9" (UID: "01c3555d-f354-45dc-9ef9-3932727a8196") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:00.721946 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.721881    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:00.722081 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.722011    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:00.722154 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.722091    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:00.722209 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.722162    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:00.722260 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.722218    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:00.722307 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:00.722292    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:00.889270 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.889229    2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:58 +0000 UTC" deadline="2028-06-30 02:41:13.122347647 +0000 UTC"
Nov 01 22:23:00.889270 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:00.889268    2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14548h18m12.233084403s"
Nov 01 22:23:01.748550 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:01.748502    2579 generic.go:356] "Generic (PLEG): container finished" podID="b6403352f0c73b775f9816057f7f4c51" containerID="ba8db321ccd67ea610f6046789be845e332cea6d27bf61555ee6024fc5580775" exitCode=0
Nov 01 22:23:01.748743 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:01.748555    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"b6403352f0c73b775f9816057f7f4c51","Type":"ContainerDied","Data":"ba8db321ccd67ea610f6046789be845e332cea6d27bf61555ee6024fc5580775"}
Nov 01 22:23:02.502606 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.502033    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:02.502606 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.502322    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:02.502606 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.502362    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:02.502606 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.502395    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret podName:ec65095d-39d7-47d4-8951-b9676b711b1d nodeName:}" failed. No retries permitted until 2026-11-01 22:23:06.502374778 +0000 UTC m=+10.250780478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret") pod "global-pull-secret-syncer-v4lzz" (UID: "ec65095d-39d7-47d4-8951-b9676b711b1d") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:02.502606 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.502508    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:02.502606 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.502581    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:06.502563525 +0000 UTC m=+10.250969219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:02.603404 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.603373    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:02.603542 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.603525    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:02.603599 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.603550    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:02.603599 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.603562    2579 projected.go:196] Error preparing data for projected volume kube-api-access-hzpx7 for pod openshift-network-diagnostics/network-check-target-gw5f9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:02.603778 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.603616    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7 podName:01c3555d-f354-45dc-9ef9-3932727a8196 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:06.60359853 +0000 UTC m=+10.352004225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzpx7" (UniqueName: "kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7") pod "network-check-target-gw5f9" (UID: "01c3555d-f354-45dc-9ef9-3932727a8196") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:02.721812 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.721778    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:02.721958 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.721896    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:02.721958 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.721956    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:02.722081 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.722016    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:02.722081 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.722062    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:02.722178 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:02.722127    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:02.757560 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.757499    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"b6403352f0c73b775f9816057f7f4c51","Type":"ContainerStarted","Data":"a474d3d21592d824c7930c9da234abf34dde99749bd6acff7efff4af1e3676a0"}
Nov 01 22:23:02.875292 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:02.874477    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" podStartSLOduration=4.874459986 podStartE2EDuration="4.874459986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:02.874254886 +0000 UTC m=+6.622660588" watchObservedRunningTime="2026-11-01 22:23:02.874459986 +0000 UTC m=+6.622865689"
Nov 01 22:23:03.775328 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:03.775228    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"6ad307b3c1f78fefabe49820df30682a","Type":"ContainerStarted","Data":"cfb73aaa62dc2c9e29049cedbe072bea6ce0e3ce43a442ceb1aa2a01645b18e3"}
Nov 01 22:23:03.778622 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:03.778595    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="4db865f85c25452559b092174ac4a6dc9042d2bda12ffc0f6fbf7b3370d41dc9" exitCode=0
Nov 01 22:23:03.778948 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:03.778837    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerDied","Data":"4db865f85c25452559b092174ac4a6dc9042d2bda12ffc0f6fbf7b3370d41dc9"}
Nov 01 22:23:04.013321 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:04.013223    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal" podStartSLOduration=6.013204123 podStartE2EDuration="6.013204123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:03.891861045 +0000 UTC m=+7.640266747" watchObservedRunningTime="2026-11-01 22:23:04.013204123 +0000 UTC m=+7.761609825"
Nov 01 22:23:04.723585 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:04.720810    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:04.723585 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:04.720983    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:04.723585 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:04.721057    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:04.723585 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:04.721180    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:04.723585 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:04.721324    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:04.727521 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:04.727480    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:04.783505 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:04.783466    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ggq5x" event={"ID":"1c493649-a226-4fdc-a95a-ba1a9de15539","Type":"ContainerStarted","Data":"0d93bb5d8aa2bdaed45b72a42bf7c67aac9f3ae9b1dbfc7305a8eeb2ea328f78"}
Nov 01 22:23:05.790866 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:05.790831    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nq2kt" event={"ID":"c14af69e-6fab-438d-82c2-4b0523bfba4a","Type":"ContainerStarted","Data":"c541693d423a3fd3ad9dcfdeb19b5b834cbdfa387725ac5f304a12fecaa282af"}
Nov 01 22:23:05.794534 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:05.794501    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw" event={"ID":"c93a44d2-477d-4501-b7ac-f681ba8f0ae9","Type":"ContainerStarted","Data":"31dcea9ba1767e9d8d6a999807f16c6a5ec7f184893dd869c7d9490888ff5b72"}
Nov 01 22:23:05.909395 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:05.909330    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ggq5x" podStartSLOduration=5.308293318 podStartE2EDuration="9.909310638s" totalImagesPullingTime="4.60101732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:04.8999251 +0000 UTC m=+8.648330802" watchObservedRunningTime="2026-11-01 22:23:05.909310638 +0000 UTC m=+9.657716341"
Nov 01 22:23:06.539563 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.539525    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:06.539743 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.539615    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:06.539817 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.539750    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:06.539817 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.539812    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.539794485 +0000 UTC m=+18.288200168 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:06.540213 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.540192    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:06.540307 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.540249    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret podName:ec65095d-39d7-47d4-8951-b9676b711b1d nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.540233459 +0000 UTC m=+18.288639153 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret") pod "global-pull-secret-syncer-v4lzz" (UID: "ec65095d-39d7-47d4-8951-b9676b711b1d") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:06.640514 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.640478    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:06.640673 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.640642    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:06.640673 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.640659    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:06.640673 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.640669    2579 projected.go:196] Error preparing data for projected volume kube-api-access-hzpx7 for pod openshift-network-diagnostics/network-check-target-gw5f9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:06.640851 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.640716    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7 podName:01c3555d-f354-45dc-9ef9-3932727a8196 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:14.640704033 +0000 UTC m=+18.389109712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzpx7" (UniqueName: "kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7") pod "network-check-target-gw5f9" (UID: "01c3555d-f354-45dc-9ef9-3932727a8196") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:06.721170 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.721137    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:06.721363 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.721243    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:06.721594 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.721575    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:06.721696 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.721668    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:06.721815 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.721801    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:06.721913 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:06.721897    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:06.930294 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.930207    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:06.931634 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:06.931431    2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:07.051148 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:07.051093    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nq2kt" podStartSLOduration=5.481856235 podStartE2EDuration="11.051073164s" totalImagesPullingTime="5.569216929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:05.909631908 +0000 UTC m=+9.658037615" watchObservedRunningTime="2026-11-01 22:23:07.051073164 +0000 UTC m=+10.799478869"
Nov 01 22:23:07.801087 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:07.801054    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-h8rr2" event={"ID":"0e10c383-d5b0-4782-9b86-85e113ae6e1c","Type":"ContainerStarted","Data":"dd13e1ba0b67ab30c60cff4985c08fb2216f1f7a6f34d71d56a86aa0bd5b9c58"}
Nov 01 22:23:07.906153 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:07.906130    2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 22:23:07.921138 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:07.920924    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-h8rr2" podStartSLOduration=3.760299944 podStartE2EDuration="11.920906742s" totalImagesPullingTime="8.160606798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:07.919990249 +0000 UTC m=+11.668395942" watchObservedRunningTime="2026-11-01 22:23:07.920906742 +0000 UTC m=+11.669312446"
Nov 01 22:23:08.701956 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.701789    2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:07.906149545Z","UUID":"7acc2654-9278-4e51-8fc0-b200b4631b26","Handler":null,"Name":"","Endpoint":""}
Nov 01 22:23:08.703799 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.703415    2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 22:23:08.703799 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.703445    2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 22:23:08.719815 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.719791    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:08.719941 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.719803    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:08.719941 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:08.719917    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:08.719941 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.719929    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:08.720103 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:08.720036    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:08.720154 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:08.720121    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:08.804468 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.804399    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="39fe41a773f0d516308f24673d7de6aded1198ac9b80a81ebaa37c9214a17173" exitCode=0
Nov 01 22:23:08.804468 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.804438    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerDied","Data":"39fe41a773f0d516308f24673d7de6aded1198ac9b80a81ebaa37c9214a17173"}
Nov 01 22:23:08.806388 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.806365    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw" event={"ID":"c93a44d2-477d-4501-b7ac-f681ba8f0ae9","Type":"ContainerStarted","Data":"1da78e4d952102dd474e29ffb10701b764f2cff3af647985343096af17143094"}
Nov 01 22:23:08.808006 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:08.807981    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6hpf6" event={"ID":"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400","Type":"ContainerStarted","Data":"8e27ba67925c6f255918d2f2f446ad954add6f1493d4b1f3f890a076bc2c189f"}
Nov 01 22:23:09.058492 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:09.058399    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6hpf6" podStartSLOduration=4.446952498 podStartE2EDuration="13.058384058s" totalImagesPullingTime="8.61143156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.057857282 +0000 UTC m=+12.806262982" watchObservedRunningTime="2026-11-01 22:23:09.058384058 +0000 UTC m=+12.806789759"
Nov 01 22:23:09.812238 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:09.811769    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nzcrt" event={"ID":"9441fd42-7d78-4065-8f63-462144f681a4","Type":"ContainerStarted","Data":"861d619daebccb7aa62ab0196f08ecd1a4d63ee4d0bc69588c54d7318e3ee852"}
Nov 01 22:23:09.815488 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:09.815458    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw" event={"ID":"c93a44d2-477d-4501-b7ac-f681ba8f0ae9","Type":"ContainerStarted","Data":"4fc991f7c91818b4e3c6673eb8ea78b83882fab844f9281b8a2ad52a34cb487d"}
Nov 01 22:23:09.928434 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:09.928341    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nzcrt" podStartSLOduration=5.496950523 podStartE2EDuration="13.928328739s" totalImagesPullingTime="8.431378216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:09.927806098 +0000 UTC m=+13.676211800" watchObservedRunningTime="2026-11-01 22:23:09.928328739 +0000 UTC m=+13.676734439"
Nov 01 22:23:10.050970 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.050910    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw" podStartSLOduration=3.84755463 podStartE2EDuration="14.050896274s" totalImagesPullingTime="10.203341644s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:10.050399046 +0000 UTC m=+13.798804752" watchObservedRunningTime="2026-11-01 22:23:10.050896274 +0000 UTC m=+13.799301974"
Nov 01 22:23:10.719596 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.719565    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:10.719752 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.719628    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:10.719752 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.719636    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:10.719752 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:10.719738    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:10.719921 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:10.719834    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:10.719982 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:10.719918    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:10.820153 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.820122    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="ad087d4d766b956209a7b12442be9ebad44c767058f09072675d3e4e67b8b959" exitCode=0
Nov 01 22:23:10.820647 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.820198    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerDied","Data":"ad087d4d766b956209a7b12442be9ebad44c767058f09072675d3e4e67b8b959"}
Nov 01 22:23:10.822280 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:10.822254    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wdllg" event={"ID":"66738f34-062c-492c-9930-7f5882b3073d","Type":"ContainerStarted","Data":"8f857b528c7d086eb230e6121c947cd30a392c18023d2a00a3b061b607a8388f"}
Nov 01 22:23:11.072949 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:11.072852    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wdllg" podStartSLOduration=4.412547739 podStartE2EDuration="15.072833401s" totalImagesPullingTime="10.660285662s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:11.072360903 +0000 UTC m=+14.820766603" watchObservedRunningTime="2026-11-01 22:23:11.072833401 +0000 UTC m=+14.821239105"
Nov 01 22:23:12.720031 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:12.719996    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:12.720031 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:12.720013    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:12.720799 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:12.720112    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:12.720799 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:12.720113    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:12.720799 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:12.720182    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:12.720799 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:12.720277    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:12.828964 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:12.828928    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="1085fbbdaedab4311338fc6ab78e3fa36cecf27091ece5c9f7e266fbb43e0436" exitCode=0
Nov 01 22:23:12.829096 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:12.828992    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerDied","Data":"1085fbbdaedab4311338fc6ab78e3fa36cecf27091ece5c9f7e266fbb43e0436"}
Nov 01 22:23:14.603838 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.603802    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:14.604347 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.603863    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:14.604347 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.603967    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:14.604347 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.603972    2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:14.604347 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.604036    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.604015387 +0000 UTC m=+34.352421075 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 22:23:14.604347 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.604054    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret podName:ec65095d-39d7-47d4-8951-b9676b711b1d nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.604045061 +0000 UTC m=+34.352450746 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret") pod "global-pull-secret-syncer-v4lzz" (UID: "ec65095d-39d7-47d4-8951-b9676b711b1d") : object "kube-system"/"original-pull-secret" not registered
Nov 01 22:23:14.705129 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.704968    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:14.705280 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.705267    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 22:23:14.705324 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.705285    2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 22:23:14.705324 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.705293    2579 projected.go:196] Error preparing data for projected volume kube-api-access-hzpx7 for pod openshift-network-diagnostics/network-check-target-gw5f9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:14.705396 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.705330    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7 podName:01c3555d-f354-45dc-9ef9-3932727a8196 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:30.705318282 +0000 UTC m=+34.453723975 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzpx7" (UniqueName: "kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7") pod "network-check-target-gw5f9" (UID: "01c3555d-f354-45dc-9ef9-3932727a8196") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 22:23:14.719589 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.719308    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:14.719589 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.719329    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:14.719589 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.719429    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:14.719589 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.719485    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:14.719589 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.719502    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:14.719589 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:14.719567    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:14.835200 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.835170    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"74c5d36671af268caefb3e07a3919d3e3e541aa25acfd415f7cb3bb4b18050b6"}
Nov 01 22:23:14.835317 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.835299    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"4a4c7d78425e16386ef4ae85890b59c68dbf82dfcaf66f89e81c3cdc047624f9"}
Nov 01 22:23:14.835371 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:14.835353    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"3b5f09746552803a863388705c20e1e30a5508bceae0f6846f305b14b270ec69"}
Nov 01 22:23:15.839796 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:15.839613    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="d332eca37360720674f1f425c77611d06b80db5d70ab10ad3a09097520286697" exitCode=0
Nov 01 22:23:15.840253 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:15.839689    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerDied","Data":"d332eca37360720674f1f425c77611d06b80db5d70ab10ad3a09097520286697"}
Nov 01 22:23:15.843132 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:15.843106    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"e9b34e764d384eb2600f6f67d8b75e29e64ea4d3812e3452ead57be8f072db5e"}
Nov 01 22:23:15.843214 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:15.843168    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"534735a62f339ebef9281450996a2f613bc57ce5acc5084a56a335da72adea7a"}
Nov 01 22:23:15.843214 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:15.843200    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"380a2e1f6a060ee76bbe9aee6904f4a8b2f9e1aecca1f4c5e97a0cbd2be3fe11"}
Nov 01 22:23:16.719921 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:16.719896    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:16.720081 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:16.719978    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:16.720154 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:16.720082    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:16.720222 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:16.720199    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:16.720277 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:16.720249    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:16.720495 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:16.720452    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:16.847095 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:16.847063    2579 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="d750a6e5bdc5983a22fcf93bcf277524518bafbccdd6e4dc762540d2aa65933a" exitCode=0
Nov 01 22:23:16.847391 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:16.847105    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerDied","Data":"d750a6e5bdc5983a22fcf93bcf277524518bafbccdd6e4dc762540d2aa65933a"}
Nov 01 22:23:17.852007 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:17.851973    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" event={"ID":"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c","Type":"ContainerStarted","Data":"0b0c22e240653914dcd46f341bf1316cfd7d237177de4e90cae8a5e93756fbd7"}
Nov 01 22:23:17.855073 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:17.855052    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"ede9fada635e01e5526ba1a1b72411e2105300bc3883b6d661438cedb7fe7baa"}
Nov 01 22:23:17.971211 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:17.971171    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-q9hwj" podStartSLOduration=8.076889895 podStartE2EDuration="20.971158754s" totalImagesPullingTime="12.894268859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:17.970658457 +0000 UTC m=+21.719064160" watchObservedRunningTime="2026-11-01 22:23:17.971158754 +0000 UTC m=+21.719564454"
Nov 01 22:23:18.719703 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:18.719676    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:18.719833 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:18.719746    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:18.719833 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:18.719751    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:18.719960 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:18.719848    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:18.719995 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:18.719955    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:18.720025 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:18.720014    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:18.753429 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:18.753408    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:18.753530 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:18.753487    2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:23:18.753940 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:18.753920    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:19.862328 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:19.862299    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" event={"ID":"589a7de9-d92b-43e5-98eb-604e083bd543","Type":"ContainerStarted","Data":"7045b8c084d86359215c1601ba575e925c8e64a115ba21b8c0662895807b4b02"}
Nov 01 22:23:19.863140 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:19.862567    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:19.863140 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:19.862640    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:19.877091 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:19.877072    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:19.881077 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:19.881063    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:20.147490 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:20.147421    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z" podStartSLOduration=7.653714283 podStartE2EDuration="23.147409327s" totalImagesPullingTime="15.493695044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 22:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:20.095915508 +0000 UTC m=+23.844321209" watchObservedRunningTime="2026-11-01 22:23:20.147409327 +0000 UTC m=+23.895815023"
Nov 01 22:23:20.721443 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:20.721411    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:20.721443 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:20.721430    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:20.721701 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:20.721437    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:20.721701 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:20.721542    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:20.721701 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:20.721634    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:20.721837 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:20.721708    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:20.864347 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:20.864327    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:21.816285 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:21.816119    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v4lzz"]
Nov 01 22:23:21.816414 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:21.816376    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:21.816469 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:21.816451    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:21.819506 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:21.819486    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r9kzv"]
Nov 01 22:23:21.819598 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:21.819579    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:21.819677 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:21.819660    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:21.867388 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:21.867359    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gw5f9"]
Nov 01 22:23:21.867747 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:21.867455    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:21.867747 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:21.867547    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:23.719482 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:23.719454    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:23.719482 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:23.719482    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:23.719864 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:23.719570    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:23.719864 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:23.719621    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:23.719864 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:23.719700    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:23.719864 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:23.719787    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:25.719742 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:25.719707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:25.720115 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:25.719707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:25.720115 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:25.719816    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v4lzz" podUID="ec65095d-39d7-47d4-8951-b9676b711b1d"
Nov 01 22:23:25.720115 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:25.719707    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:25.720115 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:25.719895    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9kzv" podUID="f9f3380f-6866-4ade-b5b8-17416d4bae89"
Nov 01 22:23:25.720115 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:25.719973    2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gw5f9" podUID="01c3555d-f354-45dc-9ef9-3932727a8196"
Nov 01 22:23:27.064439 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.064417    2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeReady"
Nov 01 22:23:27.064753 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.064513    2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 22:23:27.222596 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.222531    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qqs5h"]
Nov 01 22:23:27.225889 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.225870    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gtxmw"]
Nov 01 22:23:27.226040 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.226018    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.229891 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.229870    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.232146 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.232123    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqs5h"]
Nov 01 22:23:27.232628 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.232610    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtxmw"]
Nov 01 22:23:27.280052 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.280035    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 22:23:27.280142 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.280135    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 22:23:27.280214 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.280197    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dhrld\""
Nov 01 22:23:27.280253 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.280202    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 22:23:27.294051 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.294026    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxxpj\" (UniqueName: \"kubernetes.io/projected/3b8ce285-174b-4405-80a1-4770b4706789-kube-api-access-wxxpj\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.294140 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.294096    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b8ce285-174b-4405-80a1-4770b4706789-cert\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.384570 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.384549    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:23:27.384657 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.384599    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 22:23:27.384657 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.384641    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 22:23:27.394358 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.394335    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxpj\" (UniqueName: \"kubernetes.io/projected/3b8ce285-174b-4405-80a1-4770b4706789-kube-api-access-wxxpj\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.394416 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.394372    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-config-volume\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.394470 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.394435    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.394470 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.394459    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c686m\" (UniqueName: \"kubernetes.io/projected/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-kube-api-access-c686m\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.394562 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.394487    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b8ce285-174b-4405-80a1-4770b4706789-cert\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.394623 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.394604    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-tmp-dir\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.398001 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.397984    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b8ce285-174b-4405-80a1-4770b4706789-cert\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.401435 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.401404    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:27.452902 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.452881    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxpj\" (UniqueName: \"kubernetes.io/projected/3b8ce285-174b-4405-80a1-4770b4706789-kube-api-access-wxxpj\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.495636 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.495586    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.495636 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.495613    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c686m\" (UniqueName: \"kubernetes.io/projected/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-kube-api-access-c686m\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.495636 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.495633    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-tmp-dir\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.495805 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:27.495740    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.495805 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.495789    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-config-volume\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.495805 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:27.495801    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:27.995785966 +0000 UTC m=+31.744191652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.495915 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.495895    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-tmp-dir\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.496257 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.496240    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-config-volume\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.539377 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.539356    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:27.554082 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.554063    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c686m\" (UniqueName: \"kubernetes.io/projected/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-kube-api-access-c686m\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.632547 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.632519    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:27.719894 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.719867    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:27.719894 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.719891    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:27.720031 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.719866    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:27.774207 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774153    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 22:23:27.774207 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774168    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 22:23:27.774514 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774486    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb5v5\""
Nov 01 22:23:27.774514 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774491    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 22:23:27.774655 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774532    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:23:27.774655 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774541    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:23:27.774747 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.774670    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:23:27.809691 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.809656    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:27.915349 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.915331    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qqs5h"]
Nov 01 22:23:27.917148 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:23:27.917129    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b8ce285_174b_4405_80a1_4770b4706789.slice/crio-69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8 WatchSource:0}: Error finding container 69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8: Status 404 returned error can't find the container with id 69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8
Nov 01 22:23:27.999029 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:27.999007    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:27.999116 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:27.999098    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:27.999161 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:27.999144    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:28.999128934 +0000 UTC m=+32.747534628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:28.879766 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:28.879717    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqs5h" event={"ID":"3b8ce285-174b-4405-80a1-4770b4706789","Type":"ContainerStarted","Data":"69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8"}
Nov 01 22:23:29.005471 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:29.005446    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:29.005599 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:29.005583    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:29.005646 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:29.005637    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:31.005624276 +0000 UTC m=+34.754029956 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:30.615110 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.615072    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:30.615586 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.615147    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:30.615586 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:30.615247    2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:30.615586 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:30.615328    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:24:02.615308115 +0000 UTC m=+66.363713799 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : secret "metrics-daemon-secret" not found
Nov 01 22:23:30.617897 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.617866    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:30.715968 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.715934    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:30.718674 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.718646    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:30.734881 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.734860    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:30.745168 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.745144    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:30.902718 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:30.902652    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:31.018641 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:31.018593    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:31.018807 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:31.018778    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:31.019182 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:31.018863    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.018839991 +0000 UTC m=+38.767245692 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:31.136716 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:31.136683    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v4lzz"]
Nov 01 22:23:31.179931 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:23:31.179896    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec65095d_39d7_47d4_8951_b9676b711b1d.slice/crio-51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88 WatchSource:0}: Error finding container 51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88: Status 404 returned error can't find the container with id 51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88
Nov 01 22:23:31.186822 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:31.186798    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gw5f9"]
Nov 01 22:23:31.188518 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:23:31.188492    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c3555d_f354_45dc_9ef9_3932727a8196.slice/crio-361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba WatchSource:0}: Error finding container 361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba: Status 404 returned error can't find the container with id 361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba
Nov 01 22:23:31.890660 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:31.890584    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qqs5h" event={"ID":"3b8ce285-174b-4405-80a1-4770b4706789","Type":"ContainerStarted","Data":"5cff10fd1ec55428100f87f852a998dde2d467cc114f43334cf07859dfe3e4c2"}
Nov 01 22:23:31.891829 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:31.891800    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gw5f9" event={"ID":"01c3555d-f354-45dc-9ef9-3932727a8196","Type":"ContainerStarted","Data":"361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba"}
Nov 01 22:23:31.892806 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:31.892789    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v4lzz" event={"ID":"ec65095d-39d7-47d4-8951-b9676b711b1d","Type":"ContainerStarted","Data":"51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88"}
Nov 01 22:23:32.007716 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:32.007674    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qqs5h" podStartSLOduration=1.377104956 podStartE2EDuration="5.007663696s" totalImagesPullingTime="3.63055874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:32.006846441 +0000 UTC m=+35.755252142" watchObservedRunningTime="2026-11-01 22:23:32.007663696 +0000 UTC m=+35.756069397"
Nov 01 22:23:33.529625 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.529591    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5qrr5"]
Nov 01 22:23:33.548411 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.548385    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.607618 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.607586    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qrr5"]
Nov 01 22:23:33.633996 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.633963    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.634125 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.633999    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.634125 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.634018    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9v4t\" (UniqueName: \"kubernetes.io/projected/b088fd75-2aab-4269-b9dd-4276185761ae-kube-api-access-v9v4t\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.634247 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.634135    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b088fd75-2aab-4269-b9dd-4276185761ae-data-volume\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.634247 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.634197    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b088fd75-2aab-4269-b9dd-4276185761ae-crio-socket\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.658102 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.658078    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 22:23:33.658429 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.658294    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 22:23:33.658429 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.658320    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 22:23:33.658429 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.658392    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhsh\""
Nov 01 22:23:33.658429 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.658404    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 22:23:33.733682 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.733657    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-2zsrp-nzjfx"]
Nov 01 22:23:33.734948 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.734921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.735105 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.734959    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.735105 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.734984    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9v4t\" (UniqueName: \"kubernetes.io/projected/b088fd75-2aab-4269-b9dd-4276185761ae-kube-api-access-v9v4t\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.735105 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:33.735054    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:33.735105 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:33.735082    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:33.735340 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:33.735121    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.235104221 +0000 UTC m=+37.983509921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:33.735340 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:33.735137    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:34.235130733 +0000 UTC m=+37.983536412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:33.735340 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.735153    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b088fd75-2aab-4269-b9dd-4276185761ae-data-volume\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.735340 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.735202    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b088fd75-2aab-4269-b9dd-4276185761ae-crio-socket\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.735474 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.735377    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b088fd75-2aab-4269-b9dd-4276185761ae-crio-socket\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.735530 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.735511    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b088fd75-2aab-4269-b9dd-4276185761ae-data-volume\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.755383 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.755362    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.777338 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.777301    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:33.792078 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.791963    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2zsrp-nzjfx"]
Nov 01 22:23:33.797205 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.797184    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9v4t\" (UniqueName: \"kubernetes.io/projected/b088fd75-2aab-4269-b9dd-4276185761ae-kube-api-access-v9v4t\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:33.809544 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.809488    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-25vcq\""
Nov 01 22:23:33.809642 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.809597    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 22:23:33.836246 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.836215    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.836345 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.836252    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.836345 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.836306    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.836451 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.836405    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.897592 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.897567    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 22:23:33.937201 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.937179    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.937325 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.937218    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.937325 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.937283    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.937325 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.937312    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.937626 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.937602    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.938155 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.938132    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.938312 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.938295    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:33.954379 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.954350    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:33.997626 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:33.997599    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:34.079754 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.079501    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:34.227967 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.227932    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:34.240045 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.240012    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:34.240155 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.240065    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:34.240155 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:34.240105    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:34.240263 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:34.240166    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.240152427 +0000 UTC m=+38.988558112 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:34.240263 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:34.240185    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:34.240263 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:34.240231    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:35.240219644 +0000 UTC m=+38.988625347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:34.414308 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.414237    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:34.487833 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.487807    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-2zsrp-nzjfx"]
Nov 01 22:23:34.490477 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:23:34.490446    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87117bd_9b41_428a_8e18_f5af6fd2fd7f.slice/crio-6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8 WatchSource:0}: Error finding container 6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8: Status 404 returned error can't find the container with id 6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8
Nov 01 22:23:34.592682 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.592646    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:34.663455 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.663418    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"]
Nov 01 22:23:34.697794 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.697768    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:34.724100 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.724072    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"]
Nov 01 22:23:34.824958 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.824926    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:34.827050 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.827027    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jjz5d\""
Nov 01 22:23:34.829388 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.829365    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:34.829388 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.829365    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 22:23:34.864747 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.861011    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdxcj\" (UniqueName: \"kubernetes.io/projected/55686fc0-2a46-4674-bfc0-06754b4a02c9-kube-api-access-pdxcj\") pod \"migrator-66df5d496b-rmgcl\" (UID: \"55686fc0-2a46-4674-bfc0-06754b4a02c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:34.901850 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.901814    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx" event={"ID":"c87117bd-9b41-428a-8e18-f5af6fd2fd7f","Type":"ContainerStarted","Data":"6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8"}
Nov 01 22:23:34.961628 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:34.961557    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxcj\" (UniqueName: \"kubernetes.io/projected/55686fc0-2a46-4674-bfc0-06754b4a02c9-kube-api-access-pdxcj\") pod \"migrator-66df5d496b-rmgcl\" (UID: \"55686fc0-2a46-4674-bfc0-06754b4a02c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:35.008166 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.008134    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:35.020249 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.020217    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxcj\" (UniqueName: \"kubernetes.io/projected/55686fc0-2a46-4674-bfc0-06754b4a02c9-kube-api-access-pdxcj\") pod \"migrator-66df5d496b-rmgcl\" (UID: \"55686fc0-2a46-4674-bfc0-06754b4a02c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:35.026099 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.026080    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:35.062946 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.062921    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:35.063066 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:35.063046    2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:35.063115 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:35.063106    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.063091622 +0000 UTC m=+46.811497305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:35.264522 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.264452    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:35.264522 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.264494    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:35.264676 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:35.264544    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:35.264676 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:35.264609    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:35.264676 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:35.264617    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:37.264598127 +0000 UTC m=+41.013003806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:35.264676 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:35.264657    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:37.264642193 +0000 UTC m=+41.013047871 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:35.371640 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.371607    2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wph96"]
Nov 01 22:23:35.387188 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.387167    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.417012 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.416991    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"]
Nov 01 22:23:35.418552 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:23:35.418516    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55686fc0_2a46_4674_bfc0_06754b4a02c9.slice/crio-239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b WatchSource:0}: Error finding container 239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b: Status 404 returned error can't find the container with id 239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b
Nov 01 22:23:35.429047 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.429024    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wph96"]
Nov 01 22:23:35.441405 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.441385    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 22:23:35.441405 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.441391    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 22:23:35.441564 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.441505    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 22:23:35.441564 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.441532    2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 22:23:35.441637 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.441594    2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gs8cd\""
Nov 01 22:23:35.566913 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.566776    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-key\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.566913 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.566828    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklh5\" (UniqueName: \"kubernetes.io/projected/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-kube-api-access-dklh5\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.567054 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.566930    2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-cabundle\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.623394 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.623362    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:35.667419 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.667394    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-key\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.667561 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.667428    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklh5\" (UniqueName: \"kubernetes.io/projected/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-kube-api-access-dklh5\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.667561 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.667463    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-cabundle\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.668142 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.668123    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-cabundle\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.670310 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.670284    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-key\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.729165 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.729137    2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklh5\" (UniqueName: \"kubernetes.io/projected/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-kube-api-access-dklh5\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:35.906468 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.906395    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gw5f9" event={"ID":"01c3555d-f354-45dc-9ef9-3932727a8196","Type":"ContainerStarted","Data":"d1a6aceb0bcb8767eaac048073568f5bc71188ae17afb16265910378e1e86216"}
Nov 01 22:23:35.906808 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.906780    2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:35.908090 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:35.908063    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl" event={"ID":"55686fc0-2a46-4674-bfc0-06754b4a02c9","Type":"ContainerStarted","Data":"239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b"}
Nov 01 22:23:36.017803 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:36.017775    2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:36.343406 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:36.343329    2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:36.411506 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:36.411448    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gw5f9" podStartSLOduration=36.194154198 podStartE2EDuration="40.411427932s" totalImagesPullingTime="4.217273734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:36.028482479 +0000 UTC m=+39.776888180" watchObservedRunningTime="2026-11-01 22:23:36.411427932 +0000 UTC m=+40.159833634"
Nov 01 22:23:36.411707 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:36.411685    2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-wph96"]
Nov 01 22:23:36.415754 ip-10-0-130-144 kubenswrapper[2579]: W1101 22:23:36.415709    2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fcf42b1_1e74_4ccb_ae30_c0f893192c86.slice/crio-82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe WatchSource:0}: Error finding container 82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe: Status 404 returned error can't find the container with id 82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe
Nov 01 22:23:36.912061 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:36.912015    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wph96" event={"ID":"4fcf42b1-1e74-4ccb-ae30-c0f893192c86","Type":"ContainerStarted","Data":"82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe"}
Nov 01 22:23:37.284809 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:37.284765    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:37.284993 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:37.284818    2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:37.284993 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:37.284910    2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 01 22:23:37.284993 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:37.284951    2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:37.284993 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:37.284991    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.2849718 +0000 UTC m=+45.033377480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : configmap "kube-rbac-proxy" not found
Nov 01 22:23:37.285198 ip-10-0-130-144 kubenswrapper[2579]: E1101 22:23:37.285010    2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.285000561 +0000 UTC m=+45.033406240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:37.916086 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:37.916009    2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl" event={"ID":"55686fc0-2a46-4674-bfc0-06754b4a02c9","Type":"ContainerStarted","Data":"14a00c89f193e5b3a98a979cb72dedc9921f9dd5d482a609567b69d6a28fd1a2"}
Nov 01 22:23:38.033032 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:38.032981    2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl" podStartSLOduration=1.811581187 podStartE2EDuration="4.032964261s" totalImagesPullingTime="2.221383074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:38.032531186 +0000 UTC m=+41.780936865" watchObservedRunningTime="2026-11-01 22:23:38.032964261 +0000 UTC m=+41.781369966"
Nov 01 22:23:38.624394 ip-10-0-130-144 kubenswrapper[2579]: I1101 22:23:38.624277    2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:23:38.624437 ip-10-0-130-144 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 22:23:38.643124 ip-10-0-130-144 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 22:23:38.643288 ip-10-0-130-144 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 22:23:38.643336 ip-10-0-130-144 systemd[1]: kubelet.service: Consumed 2.268s CPU time.
Nov 01 22:23:38.653215 ip-10-0-130-144 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 22:23:38.733997 ip-10-0-130-144 kubenswrapper[4531]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:23:38.733997 ip-10-0-130-144 kubenswrapper[4531]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 22:23:38.733997 ip-10-0-130-144 kubenswrapper[4531]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:23:38.733997 ip-10-0-130-144 kubenswrapper[4531]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 22:23:38.736041 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.735973    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.736041 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736040    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.736041 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736045    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736048    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736052    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736055    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736057    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736060    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736062    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736065    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736068    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736071    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736074    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736076    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736079    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736082    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736084    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736087    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736090    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736092    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736095    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736098    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.736142 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736100    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736103    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736109    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736112    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736114    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736117    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736120    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736122    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736125    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736128    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736130    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736133    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736136    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736138    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736141    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736144    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736147    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736149    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736152    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736155    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.736662 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736157    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736160    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736163    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736165    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736168    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736170    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736173    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736176    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736179    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736182    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736184    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736187    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736189    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736194    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736197    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736200    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736203    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736206    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736209    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736212    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.737181 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736215    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736217    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736220    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736223    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736226    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736229    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736231    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736234    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736236    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736239    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736242    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736244    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736247    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736249    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736252    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736255    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736257    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736260    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736263    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736265    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.737839 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736268    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736270    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736273    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736275    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736278    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736280    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736283    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736285    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736288    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736292    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736295    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736298    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736304    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736307    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736309    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736314    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736317    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736319    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.738356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736322    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736325    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736327    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736330    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736334    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.736339    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737063    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737070    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737073    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737075    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737078    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737081    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737084    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737086    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737090    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737092    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737095    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737098    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737100    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.738822 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737103    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737106    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737108    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737111    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737113    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737117    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737120    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737123    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737126    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737128    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737133    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737135    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737138    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737141    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737143    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737146    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737149    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737151    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737154    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737158    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.739304 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737160    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737163    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737165    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737168    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737171    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737173    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737176    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737178    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737182    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737186    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737188    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737191    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737194    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737197    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737200    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737202    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737205    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737207    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737210    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.739820 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737212    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737215    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737217    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737221    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737225    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737229    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737233    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737237    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737241    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737244    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737248    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737252    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737258    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737262    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737264    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737267    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737271    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737276    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737279    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737282    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.740281 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737285    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737288    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737291    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737293    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737296    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737299    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737301    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737304    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737306    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737309    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737311    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737314    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737316    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737319    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737322    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737325    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737328    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737330    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737334    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737340    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.740795 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737344    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737348    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737352    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737356    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737360    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737364    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737369    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737373    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737378    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737382    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737386    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737390    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737395    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737399    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737935    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737942    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737946    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737949    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737952    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737954    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.741278 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737957    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737960    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737963    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737965    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737969    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737973    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737977    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737983    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737986    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737988    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737991    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737994    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.737997    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738000    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738002    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738005    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738008    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738010    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738016    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738018    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.741782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738021    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738024    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738027    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738030    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738032    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738035    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738037    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738040    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738042    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738045    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738047    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738050    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738052    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738055    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738057    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738060    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738062    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738064    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738067    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.742311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738070    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738072    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738075    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738078    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738080    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738083    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738085    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738088    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738091    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738093    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738096    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738100    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738103    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738106    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738110    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738114    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738116    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738119    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738122    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.742803 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738125    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738127    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738130    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738132    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738135    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738138    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738140    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738143    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738145    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738147    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738150    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738152    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738155    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738158    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738160    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738163    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738166    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738168    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738171    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738173    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.743286 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738176    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738179    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738181    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738184    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738188    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738190    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738195    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738199    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738202    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738205    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738207    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738210    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738212    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738215    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738217    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738220    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738222    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738225    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738227    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.743770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738230    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738232    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738235    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738330    4531 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738344    4531 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738351    4531 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738356    4531 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738361    4531 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738365    4531 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738370    4531 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738374    4531 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738377    4531 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738380    4531 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738384    4531 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738387    4531 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738390    4531 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738393    4531 flags.go:64] FLAG: --cgroup-root=""
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738397    4531 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738401    4531 flags.go:64] FLAG: --client-ca-file=""
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738403    4531 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738407    4531 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738412    4531 flags.go:64] FLAG: --cluster-domain=""
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738415    4531 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 22:23:38.744299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738418    4531 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738421    4531 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738424    4531 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738428    4531 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738431    4531 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738435    4531 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738438    4531 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738441    4531 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738444    4531 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738447    4531 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738450    4531 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738453    4531 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738459    4531 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738462    4531 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738465    4531 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738467    4531 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738471    4531 flags.go:64] FLAG: --enable-server="true"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738474    4531 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738478    4531 flags.go:64] FLAG: --event-burst="100"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738481    4531 flags.go:64] FLAG: --event-qps="50"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738484    4531 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738487    4531 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738490    4531 flags.go:64] FLAG: --eviction-hard=""
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738494    4531 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 22:23:38.745123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738497    4531 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738500    4531 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738504    4531 flags.go:64] FLAG: --eviction-soft=""
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738507    4531 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738510    4531 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738513    4531 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738517    4531 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738520    4531 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738522    4531 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738525    4531 flags.go:64] FLAG: --feature-gates=""
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738530    4531 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738533    4531 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738536    4531 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738539    4531 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738542    4531 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738545    4531 flags.go:64] FLAG: --help="false"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738548    4531 flags.go:64] FLAG: --hostname-override="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738552    4531 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738555    4531 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738558    4531 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738562    4531 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738566    4531 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738569    4531 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 22:23:38.745899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738572    4531 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738575    4531 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738578    4531 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738581    4531 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738584    4531 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738587    4531 flags.go:64] FLAG: --kube-reserved=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738590    4531 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738593    4531 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738596    4531 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738599    4531 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738602    4531 flags.go:64] FLAG: --lock-file=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738606    4531 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738609    4531 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738612    4531 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738618    4531 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738621    4531 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738624    4531 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738627    4531 flags.go:64] FLAG: --logging-format="text"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738630    4531 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738634    4531 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738637    4531 flags.go:64] FLAG: --manifest-url=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738640    4531 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738644    4531 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738647    4531 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738651    4531 flags.go:64] FLAG: --max-pods="110"
Nov 01 22:23:38.746494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738654    4531 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738657    4531 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738660    4531 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738663    4531 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738667    4531 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738670    4531 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738673    4531 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738680    4531 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738683    4531 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738686    4531 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738690    4531 flags.go:64] FLAG: --pod-cidr=""
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738692    4531 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738695    4531 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738699    4531 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738702    4531 flags.go:64] FLAG: --port="10250"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738705    4531 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738707    4531 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-08e12a394936d5a8a"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738711    4531 flags.go:64] FLAG: --qos-reserved=""
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738715    4531 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738718    4531 flags.go:64] FLAG: --register-node="true"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738721    4531 flags.go:64] FLAG: --register-with-taints=""
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738741    4531 flags.go:64] FLAG: --registry-burst="10"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738745    4531 flags.go:64] FLAG: --registry-qps="5"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738748    4531 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738751    4531 flags.go:64] FLAG: --reserved-memory=""
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738755    4531 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 22:23:38.747171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738758    4531 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738761    4531 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738764    4531 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738767    4531 flags.go:64] FLAG: --runonce="false"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738770    4531 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738773    4531 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738776    4531 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738779    4531 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738782    4531 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738785    4531 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738789    4531 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738793    4531 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738796    4531 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738799    4531 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738802    4531 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738805    4531 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738807    4531 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738811    4531 flags.go:64] FLAG: --system-cgroups=""
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738813    4531 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738819    4531 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738822    4531 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738825    4531 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738829    4531 flags.go:64] FLAG: --tls-min-version=""
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738832    4531 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738836    4531 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 22:23:38.747826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738839    4531 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738842    4531 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738845    4531 flags.go:64] FLAG: --v="2"
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738850    4531 flags.go:64] FLAG: --version="false"
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738854    4531 flags.go:64] FLAG: --vmodule=""
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738857    4531 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.738861    4531 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738993    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.738997    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739000    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739003    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739006    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739009    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739012    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739015    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739017    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739020    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739024    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739028    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739032    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739035    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.748473 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739038    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739041    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739044    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739047    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739049    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739052    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739054    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739057    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739060    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739062    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739066    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739069    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739071    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739074    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739077    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739079    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739082    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739084    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739095    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.749137 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739099    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739103    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739106    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739108    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739111    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739113    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739116    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739118    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739121    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739125    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739127    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739130    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739133    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739135    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739138    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739141    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739144    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739146    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739149    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739151    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.749703 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739154    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739157    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739159    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739163    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739166    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739168    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739171    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739174    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739176    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739179    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739181    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739184    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739187    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739190    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739193    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739195    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739198    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739200    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739203    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739205    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.750219 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739208    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739213    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739215    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739218    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739221    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739224    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739226    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739229    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739231    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739234    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739237    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739239    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739242    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739245    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739247    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739251    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739254    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739257    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739259    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739262    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.750818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739264    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739267    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739270    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739272    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739275    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739278    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739281    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739283    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739286    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739288    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739291    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739293    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739296    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.739323    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739361    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739365    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.751458 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739368    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739371    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739373    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739376    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739379    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739382    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739385    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739387    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739390    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739392    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739395    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739400    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739402    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739405    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739408    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739410    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739413    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739415    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739418    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739420    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.751890 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739423    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739426    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739429    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739431    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739434    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739436    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739439    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739441    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739444    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739446    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739450    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739453    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739456    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739459    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739461    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739464    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739466    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739470    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739475    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.752468 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739478    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739481    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739484    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739486    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739490    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739493    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739495    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739498    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739500    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739503    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739506    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739508    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739511    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739514    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739516    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739519    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739522    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739524    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739527    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.753039 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739530    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739534    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739537    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739540    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739544    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739547    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739549    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739552    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739554    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739557    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739559    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739562    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739565    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739567    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739570    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739573    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739575    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739579    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739582    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.753530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739584    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739587    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739589    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739592    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739594    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739597    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739599    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739602    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739605    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739607    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739610    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739612    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739615    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739618    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739620    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739623    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739625    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739629    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739632    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739635    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.754024 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739638    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739640    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739643    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739645    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739648    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739650    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.739653    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.743060    4531 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.743074    4531 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.743088    4531 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.743093    4531 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743150    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743155    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743159    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743162    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743165    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743168    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743171    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743174    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743176    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.754514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743179    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743182    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743185    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743191    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743195    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743200    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743203    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743206    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743209    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743211    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743215    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743217    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743220    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743223    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743226    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743228    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743231    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743233    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743236    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.755121 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743239    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743242    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743245    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743247    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743250    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743252    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743255    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743258    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743260    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743263    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743265    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743268    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743271    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743273    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743276    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743279    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743282    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743285    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743288    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743291    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.755584 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743293    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743296    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743299    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743302    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743304    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743306    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743309    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743312    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743314    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743316    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743319    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743322    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743324    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743327    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743330    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743332    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743335    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743338    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743340    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743343    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.756088 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743345    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743348    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743351    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743354    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743357    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743359    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743362    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743365    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743367    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743370    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743373    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743376    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743379    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743381    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743384    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743387    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743389    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743391    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743394    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.756573 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743397    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743399    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743402    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743404    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743408    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743413    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743417    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743419    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743422    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743425    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743427    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743430    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743432    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743435    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743437    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743440    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743443    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743445    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.757058 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743448    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.743474    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743607    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743611    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743614    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743616    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743621    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743625    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743629    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743631    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743634    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743637    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743639    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743642    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743645    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743647    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.757493 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743650    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743652    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743655    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743658    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743660    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743663    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743665    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743668    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743671    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743673    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743676    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743678    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743681    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743683    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743686    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743689    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743692    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743694    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743697    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743699    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.757917 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743701    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743704    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743707    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743710    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743712    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743715    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743717    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743720    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743722    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743741    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743744    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743747    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743750    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743752    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743755    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743758    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743760    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743763    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743765    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743768    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.758387 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743770    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743773    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743776    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743779    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743781    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743784    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743786    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743789    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743792    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743794    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743798    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743802    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743805    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743808    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743810    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743813    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743816    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743819    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743821    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.758871 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743824    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743827    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743829    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743832    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743835    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743837    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743840    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743843    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743845    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743848    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743851    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743853    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743856    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743858    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743861    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743863    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743866    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743868    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743871    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743873    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.759379 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743876    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743879    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743881    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743884    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743886    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743889    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743891    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743894    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743896    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743899    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743902    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743904    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743907    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.743929    4531 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743960    4531 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743965    4531 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 22:23:38.759973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743968    4531 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743971    4531 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743974    4531 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743976    4531 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743979    4531 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743982    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743984    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743987    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743989    4531 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743992    4531 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743994    4531 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.743997    4531 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744000    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744002    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744005    4531 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744007    4531 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744010    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744012    4531 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744015    4531 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 22:23:38.760421 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744017    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744020    4531 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744022    4531 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744025    4531 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744027    4531 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744030    4531 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744032    4531 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744035    4531 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744038    4531 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744041    4531 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744043    4531 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744046    4531 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744048    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744051    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744055    4531 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744058    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744061    4531 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744064    4531 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744066    4531 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744069    4531 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 22:23:38.761227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744071    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744073    4531 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744076    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744079    4531 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744081    4531 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744084    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744086    4531 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744089    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744091    4531 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744094    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744101    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744104    4531 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744107    4531 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744110    4531 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744113    4531 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744115    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744118    4531 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744120    4531 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744123    4531 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 22:23:38.762018 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744126    4531 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744128    4531 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744131    4531 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744133    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744136    4531 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744139    4531 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744142    4531 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744144    4531 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744147    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744149    4531 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744152    4531 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744155    4531 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744157    4531 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744160    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744163    4531 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744165    4531 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744168    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744170    4531 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744173    4531 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 22:23:38.762632 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744175    4531 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744178    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744182    4531 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744185    4531 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744190    4531 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744193    4531 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744197    4531 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744200    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744203    4531 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744205    4531 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744208    4531 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744210    4531 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744213    4531 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744215    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744218    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744221    4531 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744224    4531 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744226    4531 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744229    4531 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 22:23:38.763269 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744231    4531 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744235    4531 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744237    4531 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744240    4531 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744242    4531 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744245    4531 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744248    4531 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:38.744250    4531 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.744295    4531 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.745966    4531 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.746012    4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.746189    4531 server.go:1014] "Starting client certificate rotation"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.746327    4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.746365    4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 22:17:56 +0000 UTC" deadline="2028-06-17 10:47:58.873153857 +0000 UTC"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.746404    4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14244h24m20.126751795s"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.746618    4531 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:23:38.763757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.747660    4531 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 22:23:38.764153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.748889    4531 log.go:25] "Validated CRI v1 runtime API"
Nov 01 22:23:38.764153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.750417    4531 log.go:25] "Validated CRI v1 image API"
Nov 01 22:23:38.764153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.751140    4531 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 22:23:38.764153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.752479    4531 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 910a2775-8e62-465c-b996-6049289d113e:/dev/nvme0n1p3 ec8b3902-7669-4b82-a531-50a3ef46780e:/dev/nvme0n1p4]
Nov 01 22:23:38.764266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.752497    4531 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba/userdata/shm major:0 minor:210 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01c3555d-f354-45dc-9ef9-3932727a8196/volumes/kubernetes.io~projected/kube-api-access-hzpx7:{mountpoint:/var/lib/kubelet/pods/01c3555d-f354-45dc-9ef9-3932727a8196/volumes/kubernetes.io~projected/kube-api-access-hzpx7 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0e10c383-d5b0-4782-9b86-85e113ae6e1c/volumes/kubernetes.io~projected/kube-api-access-9vbmf:{mountpoint:/var/lib/kubelet/pods/0e10c383-d5b0-4782-9b86-85e113ae6e1c/volumes/kubernetes.io~projected/kube-api-access-9vbmf major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c493649-a226-4fdc-a95a-ba1a9de15539/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1c493649-a226-4fdc-a95a-ba1a9de15539/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b8ce285-174b-4405-80a1-4770b4706789/volumes/kubernetes.io~projected/kube-api-access-wxxpj:{mountpoint:/var/lib/kubelet/pods/3b8ce285-174b-4405-80a1-4770b4706789/volumes/kubernetes.io~projected/kube-api-access-wxxpj major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b8ce285-174b-4405-80a1-4770b4706789/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/3b8ce285-174b-4405-80a1-4770b4706789/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fcf42b1-1e74-4ccb-ae30-c0f893192c86/volumes/kubernetes.io~projected/kube-api-access-dklh5:{mountpoint:/var/lib/kubelet/pods/4fcf42b1-1e74-4ccb-ae30-c0f893192c86/volumes/kubernetes.io~projected/kube-api-access-dklh5 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4fcf42b1-1e74-4ccb-ae30-c0f893192c86/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/4fcf42b1-1e74-4ccb-ae30-c0f893192c86/volumes/kubernetes.io~secret/signing-key major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/55686fc0-2a46-4674-bfc0-06754b4a02c9/volumes/kubernetes.io~projected/kube-api-access-pdxcj:{mountpoint:/var/lib/kubelet/pods/55686fc0-2a46-4674-bfc0-06754b4a02c9/volumes/kubernetes.io~projected/kube-api-access-pdxcj major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volumes/kubernetes.io~projected/kube-api-access-wxzlm:{mountpoint:/var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volumes/kubernetes.io~projected/kube-api-access-wxzlm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~projected/kube-api-access-qbnvw:{mountpoint:/var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~projected/kube-api-access-qbnvw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400/volumes/kubernetes.io~projected/kube-api-access-5r9ps:{mountpoint:/var/lib/kubelet/pods/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400/volumes/kubernetes.io~projected/kube-api-access-5r9ps major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9441fd42-7d78-4065-8f63-462144f681a4/volumes/kubernetes.io~projected/kube-api-access-6v9hs:{mountpoint:/var/lib/kubelet/pods/9441fd42-7d78-4065-8f63-462144f681a4/volumes/kubernetes.io~projected/kube-api-access-6v9hs major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b088fd75-2aab-4269-b9dd-4276185761ae/volumes/kubernetes.io~projected/kube-api-access-v9v4t:{mountpoint:/var/lib/kubelet/pods/b088fd75-2aab-4269-b9dd-4276185761ae/volumes/kubernetes.io~projected/kube-api-access-v9v4t major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c14af69e-6fab-438d-82c2-4b0523bfba4a/volumes/kubernetes.io~projected/kube-api-access-trbxq:{mountpoint:/var/lib/kubelet/pods/c14af69e-6fab-438d-82c2-4b0523bfba4a/volumes/kubernetes.io~projected/kube-api-access-trbxq major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c87117bd-9b41-428a-8e18-f5af6fd2fd7f/volumes/kubernetes.io~projected/kube-api-access-zlnjk:{mountpoint:/var/lib/kubelet/pods/c87117bd-9b41-428a-8e18-f5af6fd2fd7f/volumes/kubernetes.io~projected/kube-api-access-zlnjk major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c93a44d2-477d-4501-b7ac-f681ba8f0ae9/volumes/kubernetes.io~projected/kube-api-access-kjljb:{mountpoint:/var/lib/kubelet/pods/c93a44d2-477d-4501-b7ac-f681ba8f0ae9/volumes/kubernetes.io~projected/kube-api-access-kjljb major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c/volumes/kubernetes.io~projected/kube-api-access-fksbz:{mountpoint:/var/lib/kubelet/pods/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c/volumes/kubernetes.io~projected/kube-api-access-fksbz major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec65095d-39d7-47d4-8951-b9676b711b1d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/ec65095d-39d7-47d4-8951-b9676b711b1d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1473442-fc77-41b5-8d1b-4c2d8aa380b3/volumes/kubernetes.io~projected/kube-api-access-c686m:{mountpoint:/var/lib/kubelet/pods/f1473442-fc77-41b5-8d1b-4c2d8aa380b3/volumes/kubernetes.io~projected/kube-api-access-c686m major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9f3380f-6866-4ade-b5b8-17416d4bae89/volumes/kubernetes.io~projected/kube-api-access-jt6df:{mountpoint:/var/lib/kubelet/pods/f9f3380f-6866-4ade-b5b8-17416d4bae89/volumes/kubernetes.io~projected/kube-api-access-jt6df major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/55db1fcf88291008f808b8abfdd096f4a10979216399311db5d59a4ffb234a70/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/5207c5acaaddefe68b649b567b3b3a511c813dac24644d4e81d14c4bad44169b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/03992c21ad0433e0dce5c745374294327956fba0e7f874a94ecd2970cab3956a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/86e9cc6a1012425842cdd4187b922d4465bf7855e7ee3d7b7ffafa45b42c91cf/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/42c562d6989a052ca25a94cb9dcf9647e9be4686d978ca0532022a283a4d2ac9/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/d470900a03e4dce06ef09c542faf4274ce42528ea108edc0fc5362f960d1258c/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/09b4c8f7c24b0d2abe7f8142c3d36e40a7eaecc42563f5825153e022dbcd9caf/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/ebcff358463a9f290340ed81b4d5fe998bcd07c48aba2965f9d520421bf76624/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/99076163bc3c4374f1846cd58764b6a55d3a61ffb37bb2a25f4dad7dc035225b/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/96bccb0ddd8bffcf7ad30d1c30ed38bba8cfadb4140263dfc0748afbff582575/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/f167db34b8d43ae7867c1cf2d90ab33af6e61662b72e16e1413610fad66d4443/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/48dc024e72288deff48693bed369805e5f0da4b90ca90e30bab1bfcc9edfaf86/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5aae8e076bb94d8b065bdd577231a816f65ef97ea46c9ee7d898344abe0c59d3/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/02ed57a09fffbd92e061319658a90ceb57c267fc0ad799669825e24ff2ed135d/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1baf29a245af8631fa33bf9f53298bf5827b09fecfe0593eb137645f901fc4fc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4fecdedefb97ba07291b0cee8cf97077f0e0e4e08470bd032420a823a73e2539/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/565a2ca6b4012fad4238e867a9eea38bcc53bc5df6009b07d25de7b95836f50c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/e26ad4deac74f1513414b01103c449547fabe9f1ac3ae9164bad7a9314a98565/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/cb560ed8aba5a46ae96b5dd66332cc1e74540fedaee41434b6f1a6532e70011d/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/cb124a629d2eb8894df5c8fb95a866b1c2419be0376e10a1171fac937b54eb77/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/5783dadcb70dca3ea91a0a027bd42851f16173626d3e8b84b1432cb2fca49b33/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/b5e4e62ca5990dab76d6e3ef84bdd961ff8552bf83036d367cb66538670c3f61/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/1ea52d13701628f25bcec1026681ecf9291f99e92db9efca5b8ae6bd06236069/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/38a0f36570e7350c42451a186db92cfebdeba2bc80856021c68d5247e8cb24f7/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/95569c455a1792094e4c6d13c28fd2e5a47a223ef29ca21ea2b367230a212512/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/e9af4c2a833fa32c25d20c7b89e444122704c26ba39898459f382d592bc15d31/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/57baaf262c92ca78ba3aa42001565ff646c78c215c66e764eae26f9612eff459/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/7bfc720ecc82a7830219c9da8f67ab899768b4c6f153f811fe480fb9a8d89893/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/d1d4dfeced6e79816347467b7b4428e6212dc46b8a2f42548a089c32650996af/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/fb12ceceb662aa5b7bf3824813dd51c7c58a6c1c51d3abda84d86ccfb05367ff/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/4d7a50af010d836cb7910a26298b7fb7b00d3fd2b4473bbfe6166ab809cc0d5a/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/52b277269f3267c429c1e43175392e409424076c72322b4bdf781be1318e696d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/48ebabef8a7446e6fbeb52a841e561f1ab887cce904742cb7c9777a6d0807af4/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/30c0637eee5563ee8accd939b128ae9faf0bb7da6397228b0db88469b1551e1a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/662ed002b420118c2e49195bcb4d0a05da1c1d14c4c0d214728aeab23c384932/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5e0f3d37fa3d662e493f615b2529c824f4affdb742362b5489a60d92b623e349/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/a50618dc9f5ed7d3029f9dbba09a6ea9a781175f7ee91acd820f2aeacbf260b6/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/37b7c893ef2c6eb9d182f5e26b49832b16ddea12dec5ec67692262f81921eca5/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/c0dcf637ec391646d2db7543df1b7284c61e8d9c41a2aaad494745627a59964e/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/d7426e9a52af8d08bd7f6de90a5138ebe0009f2e6e85b2b3f941ac6564f17a91/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/924b8be1879e3ba67f5737eba3958f5f1707caea624daeb03cb9e2f0ecfa6ca8/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/25a28a18aaadc3555375995d88c08fe4729aac088a1a94668853d0c21fa6e90b/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.759208    4531 manager.go:217] Machine: {Timestamp:2026-11-01 22:23:38.758744714 +0000 UTC m=+0.076394707 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098889 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26f4f7aed77ba0380b8f48a6c0615b SystemUUID:ec26f4f7-aed7-7ba0-380b-8f48a6c0615b BootID:3fddff22-5a75-4577-af8b-47b9b183ac1f Filesystems:[{Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volumes/kubernetes.io~projected/kube-api-access-wxzlm DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9441fd42-7d78-4065-8f63-462144f681a4/volumes/kubernetes.io~projected/kube-api-access-6v9hs DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400/volumes/kubernetes.io~projected/kube-api-access-5r9ps DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/90568667fcc50f65096a0d3cf7aa02bed3c2ea35ba79e975bc119e9f58977b11/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec65095d-39d7-47d4-8951-b9676b711b1d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0e10c383-d5b0-4782-9b86-85e113ae6e1c/volumes/kubernetes.io~projected/kube-api-access-9vbmf DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dff2a8346ff46aa124bb0ec888d58650bd322df413fd539a44686edf78080370/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b8ce285-174b-4405-80a1-4770b4706789/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/239f03bf2c1e2c1bf5b0bb3475dcea2da15e0d247299cd0f8d93b492e2e1b07b/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f9f3380f-6866-4ade-b5b8-17416d4bae89/volumes/kubernetes.io~projected/kube-api-access-jt6df DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bff5324a4bcbe15080e20f87c1afdb4ab81a863f1a15576848fcf83dfc6b53d/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1473442-fc77-41b5-8d1b-4c2d8aa380b3/volumes/kubernetes.io~projected/kube-api-access-c686m DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~projected/kube-api-access-qbnvw DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fcf42b1-1e74-4ccb-ae30-c0f893192c86/volumes/kubernetes.io~projected/kube-api-access-dklh5 DeviceMajor:0 DeviceMinor:240 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c319594fbb3dda549ee0c17898ec0b620df4e54ec3cb778b48fce10fef96d527/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/01c3555d-f354-45dc-9ef9-3932727a8196/volumes/kubernetes.io~projected/kube-api-access-hzpx7 DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/88177a8a79afe93545e5ee64fd5985278e9ac615095446625e927d0b3a533c09/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b088fd75-2aab-4269-b9dd-4276185761ae/volumes/kubernetes.io~projected/kube-api-access-v9v4t DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/55686fc0-2a46-4674-bfc0-06754b4a02c9/volumes/kubernetes.io~projected/kube-api-access-pdxcj DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4fcf42b1-1e74-4ccb-ae30-c0f893192c86/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/69dd8933da7b62b69515aed2bba6e85f60863ceff6299cd88696ecc5e91e00a8/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c87117bd-9b41-428a-8e18-f5af6fd2fd7f/volumes/kubernetes.io~projected/kube-api-access-zlnjk DeviceMajor:0 DeviceMinor:223 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/589a7de9-d92b-43e5-98eb-604e083bd543/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e163982e74fc576f899166a60dfe4dbd00bacd46da86efab66601e2e9966263/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51ede74528a18d56425401ef4ce807a30b797bc003e10e4bb69f758ce3f34b88/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1c493649-a226-4fdc-a95a-ba1a9de15539/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c93a44d2-477d-4501-b7ac-f681ba8f0ae9/volumes/kubernetes.io~projected/kube-api-access-kjljb DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c/volumes/kubernetes.io~projected/kube-api-access-fksbz DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aef153cff3a7398f443aa1c5b3e659ce4ed36d348161e09c4be956d38308d3ad/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3b8ce285-174b-4405-80a1-4770b4706789/volumes/kubernetes.io~projected/kube-api-access-wxxpj DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32f9ce276fb1295126e2c05279c4224a8415a6ecd0384b29216e33e3d41a0b0f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/361b510bb85c27a634f903f201020146ca3da4e891862732cd235902a17fe1ba/userdata/shm DeviceMajor:0 DeviceMinor:210 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/66738f34-062c-492c-9930-7f5882b3073d/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c14af69e-6fab-438d-82c2-4b0523bfba4a/volumes/kubernetes.io~projected/kube-api-access-trbxq DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82082d7849c0a70c7cb0aed3391b014bd58ccd6469ded4a36554133ed513edbe/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:239f03bf2c1e2c1 MacAddress:c2:7b:f4:2c:24:cf Speed:10000 Mtu:8901} {Name:361b510bb85c27a MacAddress:c6:76:5b:10:02:30 Speed:10000 Mtu:8901} {Name:51ede74528a18d5 MacAddress:8e:af:94:6a:de:70 Speed:10000 Mtu:8901} {Name:69dd8933da7b62b MacAddress:c6:49:79:a1:5a:22 Speed:10000 Mtu:8901} {Name:6ea2e98cd54edd8 MacAddress:8e:56:e0:d3:87:30 Speed:10000 Mtu:8901} {Name:82082d7849c0a70 MacAddress:b2:25:c5:ed:71:ab Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:ef:ca:6c:6f Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:9a:cb:e8:9d:d0 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:ef:ca:6c:6f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:be:1d:91:f3:f8:bc Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6e:6e:96:1c:c3:d4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.759570    4531 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.759629    4531 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760089    4531 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760111    4531 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-144.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760446    4531 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760459    4531 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760473    4531 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760491    4531 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760581    4531 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.760907    4531 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.761083    4531 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.761119    4531 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.761164    4531 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.761283    4531 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.761386    4531 kubelet.go:405] "Adding apiserver pod source"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.762142    4531 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.762686    4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.762696    4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:23:38.764416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.763167    4531 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 22:23:38.771366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771345    4531 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 22:23:38.771366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771363    4531 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 22:23:38.771474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771460    4531 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 22:23:38.771646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771635    4531 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 22:23:38.771686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771665    4531 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 22:23:38.771824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771809    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 22:23:38.771856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771838    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 22:23:38.771856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771848    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 22:23:38.771856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771853    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771859    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771866    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771871    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771877    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771883    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771889    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771902    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771912    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 22:23:38.771936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771936    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 22:23:38.772172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.771945    4531 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 22:23:38.775167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.775153    4531 server.go:1263] "Started kubelet"
Nov 01 22:23:38.775241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.775211    4531 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 22:23:38.775333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.775272    4531 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 22:23:38.775379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.775360    4531 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 22:23:38.776344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.775789    4531 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 22:23:38.776427 ip-10-0-130-144 systemd[1]: Started Kubernetes Kubelet.
Nov 01 22:23:38.777064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.776933    4531 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.780705    4531 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.780739    4531 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.780805    4531 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 22:17:58 +0000 UTC" deadline="2028-07-02 10:14:46.052923801 +0000 UTC"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.780840    4531 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14603h51m7.272087967s"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.780884    4531 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.781033    4531 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 22:23:38.781045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.781043    4531 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 22:23:38.781409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.781080    4531 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 22:23:38.782520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782491    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b8ce285-174b-4405-80a1-4770b4706789" volumeName="kubernetes.io/secret/3b8ce285-174b-4405-80a1-4770b4706789-cert" seLinuxMountContext=""
Nov 01 22:23:38.782520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782516    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="55686fc0-2a46-4674-bfc0-06754b4a02c9" volumeName="kubernetes.io/projected/55686fc0-2a46-4674-bfc0-06754b4a02c9-kube-api-access-pdxcj" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782526    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="589a7de9-d92b-43e5-98eb-604e083bd543" volumeName="kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782541    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="589a7de9-d92b-43e5-98eb-604e083bd543" volumeName="kubernetes.io/projected/589a7de9-d92b-43e5-98eb-604e083bd543-kube-api-access-wxzlm" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782548    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b088fd75-2aab-4269-b9dd-4276185761ae" volumeName="kubernetes.io/empty-dir/b088fd75-2aab-4269-b9dd-4276185761ae-data-volume" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782555    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" volumeName="kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782573    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01c3555d-f354-45dc-9ef9-3932727a8196" volumeName="kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782580    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e10c383-d5b0-4782-9b86-85e113ae6e1c" volumeName="kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cni-binary-copy" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782590    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fcf42b1-1e74-4ccb-ae30-c0f893192c86" volumeName="kubernetes.io/projected/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-kube-api-access-dklh5" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782604    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="589a7de9-d92b-43e5-98eb-604e083bd543" volumeName="kubernetes.io/secret/589a7de9-d92b-43e5-98eb-604e083bd543-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782611    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c493649-a226-4fdc-a95a-ba1a9de15539" volumeName="kubernetes.io/secret/1c493649-a226-4fdc-a95a-ba1a9de15539-agent-certs" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782618    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" volumeName="kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782641    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c93a44d2-477d-4501-b7ac-f681ba8f0ae9" volumeName="kubernetes.io/projected/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kube-api-access-kjljb" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782648    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" volumeName="kubernetes.io/projected/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-kube-api-access-fksbz" seLinuxMountContext=""
Nov 01 22:23:38.782656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782657    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec65095d-39d7-47d4-8951-b9676b711b1d" volumeName="kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782670    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1473442-fc77-41b5-8d1b-4c2d8aa380b3" volumeName="kubernetes.io/configmap/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-config-volume" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782678    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e10c383-d5b0-4782-9b86-85e113ae6e1c" volumeName="kubernetes.io/projected/0e10c383-d5b0-4782-9b86-85e113ae6e1c-kube-api-access-9vbmf" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782685    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c493649-a226-4fdc-a95a-ba1a9de15539" volumeName="kubernetes.io/configmap/1c493649-a226-4fdc-a95a-ba1a9de15539-konnectivity-ca" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782703    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fcf42b1-1e74-4ccb-ae30-c0f893192c86" volumeName="kubernetes.io/configmap/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-cabundle" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782715    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c668ab-f0d3-45ee-a5c0-6f37c2ff8400" volumeName="kubernetes.io/projected/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-kube-api-access-5r9ps" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782750    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9441fd42-7d78-4065-8f63-462144f681a4" volumeName="kubernetes.io/projected/9441fd42-7d78-4065-8f63-462144f681a4-kube-api-access-6v9hs" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782815    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" volumeName="kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782822    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" volumeName="kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782828    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1473442-fc77-41b5-8d1b-4c2d8aa380b3" volumeName="kubernetes.io/empty-dir/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-tmp-dir" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782842    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b8ce285-174b-4405-80a1-4770b4706789" volumeName="kubernetes.io/projected/3b8ce285-174b-4405-80a1-4770b4706789-kube-api-access-wxxpj" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782853    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4fcf42b1-1e74-4ccb-ae30-c0f893192c86" volumeName="kubernetes.io/secret/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-key" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782863    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="589a7de9-d92b-43e5-98eb-604e083bd543" volumeName="kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-env-overrides" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782878    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66738f34-062c-492c-9930-7f5882b3073d" volumeName="kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-tmp" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782896    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9441fd42-7d78-4065-8f63-462144f681a4" volumeName="kubernetes.io/configmap/9441fd42-7d78-4065-8f63-462144f681a4-iptables-alerter-script" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782914    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b088fd75-2aab-4269-b9dd-4276185761ae" volumeName="kubernetes.io/projected/b088fd75-2aab-4269-b9dd-4276185761ae-kube-api-access-v9v4t" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782941    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" volumeName="kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782968    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" volumeName="kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-binary-copy" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.782980    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1473442-fc77-41b5-8d1b-4c2d8aa380b3" volumeName="kubernetes.io/projected/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-kube-api-access-c686m" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783002    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="589a7de9-d92b-43e5-98eb-604e083bd543" volumeName="kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-config" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783032    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66738f34-062c-492c-9930-7f5882b3073d" volumeName="kubernetes.io/projected/66738f34-062c-492c-9930-7f5882b3073d-kube-api-access-qbnvw" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783044    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14af69e-6fab-438d-82c2-4b0523bfba4a" volumeName="kubernetes.io/configmap/c14af69e-6fab-438d-82c2-4b0523bfba4a-serviceca" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783067    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" volumeName="kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783079    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9f3380f-6866-4ade-b5b8-17416d4bae89" volumeName="kubernetes.io/projected/f9f3380f-6866-4ade-b5b8-17416d4bae89-kube-api-access-jt6df" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783090    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c14af69e-6fab-438d-82c2-4b0523bfba4a" volumeName="kubernetes.io/projected/c14af69e-6fab-438d-82c2-4b0523bfba4a-kube-api-access-trbxq" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783101    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0e10c383-d5b0-4782-9b86-85e113ae6e1c" volumeName="kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-daemon-config" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783112    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66738f34-062c-492c-9930-7f5882b3073d" volumeName="kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-etc-tuned" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783123    4531 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91c668ab-f0d3-45ee-a5c0-6f37c2ff8400" volumeName="kubernetes.io/empty-dir/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-tmp-dir" seLinuxMountContext=""
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783137    4531 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 22:23:38.783375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783145    4531 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783459    4531 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783482    4531 factory.go:55] Registering systemd factory
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783492    4531 factory.go:223] Registration of the systemd container factory successfully
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783898    4531 factory.go:153] Registering CRI-O factory
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783912    4531 factory.go:223] Registration of the crio container factory successfully
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783927    4531 factory.go:103] Registering Raw factory
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.783941    4531 manager.go:1199] Started watching for new ooms in manager
Nov 01 22:23:38.784538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.784361    4531 manager.go:319] Starting recovery of all containers
Nov 01 22:23:38.806912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.806876    4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 22:23:38.808281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.808261    4531 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 22:23:38.808346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.808287    4531 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 22:23:38.808346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.808311    4531 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 22:23:38.808435 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.808367    4531 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 22:23:38.817897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.817861    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="d750a6e5bdc5983a22fcf93bcf277524518bafbccdd6e4dc762540d2aa65933a" exitCode=0
Nov 01 22:23:38.817967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.817899    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="d332eca37360720674f1f425c77611d06b80db5d70ab10ad3a09097520286697" exitCode=0
Nov 01 22:23:38.817967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.817908    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="1085fbbdaedab4311338fc6ab78e3fa36cecf27091ece5c9f7e266fbb43e0436" exitCode=0
Nov 01 22:23:38.817967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.817916    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="ad087d4d766b956209a7b12442be9ebad44c767058f09072675d3e4e67b8b959" exitCode=0
Nov 01 22:23:38.817967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.817921    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="39fe41a773f0d516308f24673d7de6aded1198ac9b80a81ebaa37c9214a17173" exitCode=0
Nov 01 22:23:38.817967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.817928    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c" containerID="4db865f85c25452559b092174ac4a6dc9042d2bda12ffc0f6fbf7b3370d41dc9" exitCode=0
Nov 01 22:23:38.845336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.845241    4531 generic.go:356] "Generic (PLEG): container finished" podID="b6403352f0c73b775f9816057f7f4c51" containerID="ba8db321ccd67ea610f6046789be845e332cea6d27bf61555ee6024fc5580775" exitCode=0
Nov 01 22:23:38.851624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.851608    4531 manager.go:324] Recovery completed
Nov 01 22:23:38.863375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.863360    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.864302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.864282    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.864394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.864317    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.864394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.864331    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.865615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865599    4531 cpu_manager.go:226] "Starting" policy="none"
Nov 01 22:23:38.865615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865614    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.865704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865615    4531 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 22:23:38.865704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865678    4531 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 22:23:38.865829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865815    4531 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 22:23:38.865860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865829    4531 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 22:23:38.865895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865844    4531 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 22:23:38.865895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865884    4531 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 22:23:38.865895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865892    4531 policy_none.go:50] "Start"
Nov 01 22:23:38.865972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865906    4531 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 22:23:38.865972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.865920    4531 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 22:23:38.866087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.866077    4531 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 22:23:38.866119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.866087    4531 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 22:23:38.866119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.866093    4531 policy_none.go:44] "Start"
Nov 01 22:23:38.866488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.866471    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.866536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.866504    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.866536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.866519    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.870452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.870430    4531 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 22:23:38.873026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873015    4531 manager.go:341] "Starting Device Plugin manager"
Nov 01 22:23:38.873069 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.873038    4531 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 22:23:38.873069 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873045    4531 server.go:87] "Starting device plugin registration server"
Nov 01 22:23:38.873350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873337    4531 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 22:23:38.873388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873351    4531 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 22:23:38.873493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873469    4531 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 22:23:38.873608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873572    4531 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 22:23:38.873673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873615    4531 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 22:23:38.873673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873623    4531 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 22:23:38.873762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.873664    4531 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T22:23:38.873605892Z","UUID":"eab1efba-f582-467c-8f20-a39e2f357b6c","Handler":null,"Name":"","Endpoint":""}
Nov 01 22:23:38.874709 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.874635    4531 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 22:23:38.874709 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.874677    4531 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-144.us-west-2.compute.internal\" not found"
Nov 01 22:23:38.875014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.874996    4531 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 22:23:38.875014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.875016    4531 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 22:23:38.908942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.908897    4531 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"]
Nov 01 22:23:38.909045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.909005    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.909717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.909703    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.909779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.909745    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.909779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.909759    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.910012 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.909979    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.910121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910105    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.910667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910643    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.910768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910671    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.910768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910682    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.910768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910647    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.910768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910716    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.910768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910752    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.910966 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.910834    4531 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:38.911016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.910991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"6ad307b3c1f78fefabe49820df30682a","Type":"ContainerStarted","Data":"cfb73aaa62dc2c9e29049cedbe072bea6ce0e3ce43a442ceb1aa2a01645b18e3"}
Nov 01 22:23:38.911072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911022    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"6ad307b3c1f78fefabe49820df30682a","Type":"ContainerStarted","Data":"05395434bc96f5a7cbd227373d15a1bc35eb03bd048284a9e583476f0501690a"}
Nov 01 22:23:38.911072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911050    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"b6403352f0c73b775f9816057f7f4c51","Type":"ContainerStarted","Data":"a474d3d21592d824c7930c9da234abf34dde99749bd6acff7efff4af1e3676a0"}
Nov 01 22:23:38.911072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911061    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"b6403352f0c73b775f9816057f7f4c51","Type":"ContainerDied","Data":"ba8db321ccd67ea610f6046789be845e332cea6d27bf61555ee6024fc5580775"}
Nov 01 22:23:38.911072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911073    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal" event={"ID":"b6403352f0c73b775f9816057f7f4c51","Type":"ContainerStarted","Data":"a7e405d29f8dd7bf300962d62263aa00e175a86506506395dae3d0964c262630"}
Nov 01 22:23:38.911241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911095    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.911785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911769    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.911846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911793    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.911846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.911802    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.911923 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:38.911912    4531 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-144.us-west-2.compute.internal\" not found" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:38.957441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.957421    4531 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.957545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.957421    4531 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.957615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.957557    4531 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 22:23:38.957615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.957558    4531 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:38.973661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.973646    4531 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 22:23:38.974392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.974379    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 22:23:38.974450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.974401    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 22:23:38.974450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.974410    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 22:23:38.974450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.974437    4531 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:38.981555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:38.981542    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.004383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.004366    4531 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 22:23:39.049049 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.049027    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:39.049128 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:39.049067    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.049128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.049082    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.057840 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.057820    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ad307b3c1f78fefabe49820df30682a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"6ad307b3c1f78fefabe49820df30682a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.057914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.057845    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.057914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.057864    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.079054 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.079035    4531 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.079124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.079080    4531 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.079455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.079440    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeNotReady"
Nov 01 22:23:39.079512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.079466    4531 setters.go:546] "Node became not ready" node="ip-10-0-130-144.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T22:23:39Z","lastTransitionTime":"2026-11-01T22:23:39Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 22:23:39.104647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.104627    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:39.104713 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:39.104660    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.107003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.106985    4531 nodeinfomanager.go:471] existing CSINode "ip-10-0-130-144.us-west-2.compute.internal" is owned by different node (oldNodeID="9670017a-f22b-4662-acc9-20d0b4634b96", newNodeID=""), cleaning up...
Nov 01 22:23:39.158458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.158409    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ad307b3c1f78fefabe49820df30682a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"6ad307b3c1f78fefabe49820df30682a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.158458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.158436    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.158458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.158454    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.158641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.158504    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6ad307b3c1f78fefabe49820df30682a-config\") pod \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"6ad307b3c1f78fefabe49820df30682a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.158641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.158504    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.158641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.158537    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/b6403352f0c73b775f9816057f7f4c51-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" (UID: \"b6403352f0c73b775f9816057f7f4c51\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.762715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.762687    4531 apiserver.go:52] "Watching apiserver"
Nov 01 22:23:39.847542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.847520    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.847781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.847623    4531 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.919110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.919091    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:39.919204 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:39.919134    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.919204 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.919093    4531 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 22:23:39.919265 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:39.919226    4531 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal"
Nov 01 22:23:39.919294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.919281    4531 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 22:23:39.919753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.919700    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-bn45z","kube-system/konnectivity-agent-ggq5x","kube-system/kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal","openshift-insights/periodic-gathering-2zsrp-nzjfx","openshift-multus/network-metrics-daemon-r9kzv","openshift-cluster-node-tuning-operator/tuned-wdllg","openshift-multus/multus-additional-cni-plugins-q9hwj","openshift-network-operator/iptables-alerter-nzcrt","openshift-multus/multus-h8rr2","openshift-service-ca/service-ca-5fffc97bdf-wph96","openshift-image-registry/node-ca-nq2kt","kube-system/global-pull-secret-syncer-v4lzz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw","openshift-dns/node-resolver-6hpf6","openshift-ingress-canary/ingress-canary-qqs5h","openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl","openshift-insights/insights-runtime-extractor-5qrr5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-144.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-gw5f9","openshift-dns/dns-default-gtxmw"]
Nov 01 22:23:39.926351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.926327    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:39.927803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.927787    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:39.928556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.928536    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:39.969657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.969634    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 22:23:39.969764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.969664    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.969817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.969785    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 22:23:39.969865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.969831    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 22:23:39.969901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.969836    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 22:23:39.969901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.969869    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-67p2k\""
Nov 01 22:23:39.970248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.970236    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-jm5kv\""
Nov 01 22:23:39.970283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.970240    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-sb5v5\""
Nov 01 22:23:39.971237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.971225    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 22:23:39.982661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:39.982641    4531 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 22:23:40.062467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062423    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vbmf\" (UniqueName: \"kubernetes.io/projected/0e10c383-d5b0-4782-9b86-85e113ae6e1c-kube-api-access-9vbmf\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.062467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062450    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-dbus\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.062467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062466    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-sys-fs\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.062664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062481    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.062664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-env-overrides\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.062664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062564    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-conf\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.062831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062666    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trbxq\" (UniqueName: \"kubernetes.io/projected/c14af69e-6fab-438d-82c2-4b0523bfba4a-kube-api-access-trbxq\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.062831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062746    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-kubelet-config\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.062831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062795    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-device-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.062831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062823    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.062996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062847    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-kubelet\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.062996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062881    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.062996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062960    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.063089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.062998    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.063089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063027    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cnibin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fksbz\" (UniqueName: \"kubernetes.io/projected/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-kube-api-access-fksbz\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.063172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063131    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.063201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063186    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c493649-a226-4fdc-a95a-ba1a9de15539-konnectivity-ca\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:40.063260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-daemon-config\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063269    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-multus-certs\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063294    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589a7de9-d92b-43e5-98eb-604e083bd543-ovn-node-metrics-cert\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.063344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063328    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.063455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063404    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.063455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063423    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.063455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063444    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-systemd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.063593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-ovn\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.063593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063483    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-config\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.063593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-tmp-dir\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.063593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063548    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1c493649-a226-4fdc-a95a-ba1a9de15539-konnectivity-ca\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:40.063789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063614    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cni-binary-copy\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063647    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-tmp-dir\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.063789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-bin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063717    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-kubelet\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063773    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-system-cni-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063796    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-var-lib-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063812    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-hosts-file\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-multus\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063888    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-etc-kubernetes\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063911    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjljb\" (UniqueName: \"kubernetes.io/projected/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kube-api-access-kjljb\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063927    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063942    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b088fd75-2aab-4269-b9dd-4276185761ae-crio-socket\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.063972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063958    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-socket-dir-parent\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063982    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-conf-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.063989    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-host\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064028    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064048    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-systemd-units\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064080    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-slash\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064099    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-kubernetes\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064123    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c686m\" (UniqueName: \"kubernetes.io/projected/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-kube-api-access-c686m\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064138    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-system-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064161    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-cabundle\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.064282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064255    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-netd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-tmp-dir\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064327    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbnvw\" (UniqueName: \"kubernetes.io/projected/66738f34-062c-492c-9930-7f5882b3073d-kube-api-access-qbnvw\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064345    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-tmp-dir\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c14af69e-6fab-438d-82c2-4b0523bfba4a-serviceca\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064491    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9hs\" (UniqueName: \"kubernetes.io/projected/9441fd42-7d78-4065-8f63-462144f681a4-kube-api-access-6v9hs\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064517    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-node-log\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064553    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-log-socket\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064575    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzlm\" (UniqueName: \"kubernetes.io/projected/589a7de9-d92b-43e5-98eb-604e083bd543-kube-api-access-wxzlm\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064590    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-k8s-cni-cncf-io\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064607    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-systemd\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064604    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c14af69e-6fab-438d-82c2-4b0523bfba4a-serviceca\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysconfig\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064644    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-netns\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064672    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-hostroot\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064692    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064714    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6df\" (UniqueName: \"kubernetes.io/projected/f9f3380f-6866-4ade-b5b8-17416d4bae89-kube-api-access-jt6df\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064746    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-bin\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064790    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-modprobe-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.064799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064813    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxxpj\" (UniqueName: \"kubernetes.io/projected/3b8ce285-174b-4405-80a1-4770b4706789-kube-api-access-wxxpj\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064939    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c14af69e-6fab-438d-82c2-4b0523bfba4a-host\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9441fd42-7d78-4065-8f63-462144f681a4-iptables-alerter-script\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.064988    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9441fd42-7d78-4065-8f63-462144f681a4-host-slash\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065041    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-netns\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065068    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065083    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-lib-modules\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065141    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-var-lib-kubelet\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065156    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b8ce285-174b-4405-80a1-4770b4706789-cert\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065182    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-config-volume\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065234    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b088fd75-2aab-4269-b9dd-4276185761ae-data-volume\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-key\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065268    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklh5\" (UniqueName: \"kubernetes.io/projected/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-kube-api-access-dklh5\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065286    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cnibin\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065299    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-sys\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065321    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdxcj\" (UniqueName: \"kubernetes.io/projected/55686fc0-2a46-4674-bfc0-06754b4a02c9-kube-api-access-pdxcj\") pod \"migrator-66df5d496b-rmgcl\" (UID: \"55686fc0-2a46-4674-bfc0-06754b4a02c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065357    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-socket-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.065491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b088fd75-2aab-4269-b9dd-4276185761ae-data-volume\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065507    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-registration-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065546    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065601    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-etc-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065629    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065661    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9v4t\" (UniqueName: \"kubernetes.io/projected/b088fd75-2aab-4269-b9dd-4276185761ae-kube-api-access-v9v4t\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065703    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-run\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065766    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-tmp\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065856    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065879    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-tmp\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065906    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c493649-a226-4fdc-a95a-ba1a9de15539-agent-certs\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:40.066046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.065959    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r9ps\" (UniqueName: \"kubernetes.io/projected/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-kube-api-access-5r9ps\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.066426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.066093    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-script-lib\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.066426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.066162    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-etc-tuned\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.066426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.066188    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.066426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.066204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1c493649-a226-4fdc-a95a-ba1a9de15539-agent-certs\") pod \"konnectivity-agent-ggq5x\" (UID: \"1c493649-a226-4fdc-a95a-ba1a9de15539\") " pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:40.066426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.066205    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.066426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.066304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/66738f34-062c-492c-9930-7f5882b3073d-etc-tuned\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.070289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.070275    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 22:23:40.072087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072064    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 22:23:40.072165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072098    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 22:23:40.072165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072137    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gs8cd\""
Nov 01 22:23:40.072279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072264    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.072446 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072431    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.072501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072432    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.072501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072457    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.072501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072477    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.072501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072488    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 22:23:40.072501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072478    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 22:23:40.072684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072539    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 22:23:40.072684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072594    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-w2xg9\""
Nov 01 22:23:40.072822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072809    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.072877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.072851    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-env-overrides\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.073772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.073752    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-n4xsx\""
Nov 01 22:23:40.073874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.073777    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.073874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.073825    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.073874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.073838    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.073965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.073873    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-daemon-config\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.073965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.073899    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hb4ll\""
Nov 01 22:23:40.076255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.076232    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-config-volume\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.076296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.076273    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.085842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.085819    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.085924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.085838    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 01 22:23:40.085924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.085866    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.085924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.085891    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xwhsh\""
Nov 01 22:23:40.086079 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.085954    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 01 22:23:40.086079 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086015    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 01 22:23:40.086193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086179    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.086295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086281    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 22:23:40.086410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086396    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 22:23:40.086563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086544    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 22:23:40.086563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086554    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.086759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086620    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 22:23:40.086759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086620    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 22:23:40.086759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086669    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.086914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086865    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.086914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086876    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 22:23:40.086914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086885    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.086914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086891    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.087098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086928    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 22:23:40.087098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.086940    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.087098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087037    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.087098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087065    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ftrlf\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087098    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-dhrld\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087100    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lmcrl\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087121    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5wj42\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087119    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-25vcq\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087131    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ljglf\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087191    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 22:23:40.087255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087200    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-jbqbs\""
Nov 01 22:23:40.087460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087270    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 01 22:23:40.087460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087329    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 01 22:23:40.087460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087343    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 22:23:40.087460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087425    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 22:23:40.087589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087508    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jl5pd\""
Nov 01 22:23:40.087622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087615    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 22:23:40.087873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087862    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jjz5d\""
Nov 01 22:23:40.087951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.087940    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j6ll7\""
Nov 01 22:23:40.093151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.093134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.094282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094262    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-config\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.094340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094284    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cni-binary-copy\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.094340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094320    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.094453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/589a7de9-d92b-43e5-98eb-604e083bd543-ovn-node-metrics-cert\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.094453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094420    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-cabundle\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:40.094453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094433    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ec65095d-39d7-47d4-8951-b9676b711b1d-original-pull-secret\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.094574 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.094490    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cni-binary-copy\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.095569 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.095553    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b8ce285-174b-4405-80a1-4770b4706789-cert\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:40.095622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.095555    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-signing-key\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:40.095622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.095606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/9441fd42-7d78-4065-8f63-462144f681a4-iptables-alerter-script\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.097223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.096861    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/589a7de9-d92b-43e5-98eb-604e083bd543-ovnkube-script-lib\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.121046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.121023    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 01 22:23:40.121370 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.121354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.125808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.125787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") pod \"periodic-gathering-2zsrp-nzjfx\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") " pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:23:40.129134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.129114    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trbxq\" (UniqueName: \"kubernetes.io/projected/c14af69e-6fab-438d-82c2-4b0523bfba4a-kube-api-access-trbxq\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.129200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.129136    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjljb\" (UniqueName: \"kubernetes.io/projected/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kube-api-access-kjljb\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.129200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.129156    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fksbz\" (UniqueName: \"kubernetes.io/projected/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-kube-api-access-fksbz\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.129381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.129359    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9hs\" (UniqueName: \"kubernetes.io/projected/9441fd42-7d78-4065-8f63-462144f681a4-kube-api-access-6v9hs\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.129420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.129367    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vbmf\" (UniqueName: \"kubernetes.io/projected/0e10c383-d5b0-4782-9b86-85e113ae6e1c-kube-api-access-9vbmf\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.132992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.132968    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c686m\" (UniqueName: \"kubernetes.io/projected/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-kube-api-access-c686m\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.132992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.132984    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6df\" (UniqueName: \"kubernetes.io/projected/f9f3380f-6866-4ade-b5b8-17416d4bae89-kube-api-access-jt6df\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:40.133135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133051    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxxpj\" (UniqueName: \"kubernetes.io/projected/3b8ce285-174b-4405-80a1-4770b4706789-kube-api-access-wxxpj\") pod \"ingress-canary-qqs5h\" (UID: \"3b8ce285-174b-4405-80a1-4770b4706789\") " pod="openshift-ingress-canary/ingress-canary-qqs5h"
Nov 01 22:23:40.133135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133054    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r9ps\" (UniqueName: \"kubernetes.io/projected/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-kube-api-access-5r9ps\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.133135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133084    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbnvw\" (UniqueName: \"kubernetes.io/projected/66738f34-062c-492c-9930-7f5882b3073d-kube-api-access-qbnvw\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.133135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133115    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdxcj\" (UniqueName: \"kubernetes.io/projected/55686fc0-2a46-4674-bfc0-06754b4a02c9-kube-api-access-pdxcj\") pod \"migrator-66df5d496b-rmgcl\" (UID: \"55686fc0-2a46-4674-bfc0-06754b4a02c9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-rmgcl"
Nov 01 22:23:40.133325 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133305    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpx7\" (UniqueName: \"kubernetes.io/projected/01c3555d-f354-45dc-9ef9-3932727a8196-kube-api-access-hzpx7\") pod \"network-check-target-gw5f9\" (UID: \"01c3555d-f354-45dc-9ef9-3932727a8196\") " pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:40.133499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133481    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9v4t\" (UniqueName: \"kubernetes.io/projected/b088fd75-2aab-4269-b9dd-4276185761ae-kube-api-access-v9v4t\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.133839 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.133824    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzlm\" (UniqueName: \"kubernetes.io/projected/589a7de9-d92b-43e5-98eb-604e083bd543-kube-api-access-wxzlm\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.136562 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.136546    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklh5\" (UniqueName: \"kubernetes.io/projected/4fcf42b1-1e74-4ccb-ae30-c0f893192c86-kube-api-access-dklh5\") pod \"service-ca-5fffc97bdf-wph96\" (UID: \"4fcf42b1-1e74-4ccb-ae30-c0f893192c86\") " pod="openshift-service-ca/service-ca-5fffc97bdf-wph96"
Nov 01 22:23:40.167048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-etc-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167056    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167072    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-run\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167096    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167119    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167127    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-etc-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167132    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167143    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-dbus\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-run\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.167193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167185    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-sys-fs\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167211    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167214    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-dbus\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167218    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.167230    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167257    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167263    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-conf\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167232    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-sys-fs\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167282    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-kubelet-config\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.167320    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.667294619 +0000 UTC m=+1.984944615 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167320    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ec65095d-39d7-47d4-8951-b9676b711b1d-kubelet-config\") pod \"global-pull-secret-syncer-v4lzz\" (UID: \"ec65095d-39d7-47d4-8951-b9676b711b1d\") " pod="kube-system/global-pull-secret-syncer-v4lzz"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167361    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-device-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-conf\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167391    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167417    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-kubelet\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167432    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-device-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167456    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-kubelet\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167470    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167481    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.167502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167511    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cnibin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167533    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.167540    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167558    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-multus-certs\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167585    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-systemd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.167603    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.667589 +0000 UTC m=+1.985238983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-cnibin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167620    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167627    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-systemd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167635    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-multus-certs\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167637    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-ovn\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167669    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-bin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167670    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-run-ovn\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167684    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-kubelet\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167706    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-system-cni-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167735    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-bin\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167741    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-kubelet\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-var-lib-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167762    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-system-cni-dir\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167776    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-hosts-file\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167798    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-multus\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167812    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-etc-kubernetes\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167827    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b088fd75-2aab-4269-b9dd-4276185761ae-crio-socket\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167826    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/91c668ab-f0d3-45ee-a5c0-6f37c2ff8400-hosts-file\") pod \"node-resolver-6hpf6\" (UID: \"91c668ab-f0d3-45ee-a5c0-6f37c2ff8400\") " pod="openshift-dns/node-resolver-6hpf6"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167778    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-var-lib-openvswitch\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167845    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-socket-dir-parent\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167867    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-conf-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167871    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-etc-kubernetes\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167871    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-var-lib-cni-multus\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167884    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b088fd75-2aab-4269-b9dd-4276185761ae-crio-socket\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167901    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-host\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167906    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-socket-dir-parent\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167913    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-multus-conf-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167933    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b088fd75-2aab-4269-b9dd-4276185761ae-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167949    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-systemd-units\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167953    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-host\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167972    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-slash\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167980    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-systemd-units\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.167991    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-kubernetes\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168006    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-system-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168011    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-slash\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168019    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-netd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168042    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-kubernetes\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168044    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168053    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysctl-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168068    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-netd\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168068    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-node-log\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168092    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-node-log\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168093    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-log-socket\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168106    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-etc-selinux\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168116    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-k8s-cni-cncf-io\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-system-cni-dir\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168110    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-log-socket\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168135    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-systemd\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168145    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-k8s-cni-cncf-io\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168150    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysconfig\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168161    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-systemd\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168176    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-sysconfig\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168181    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-netns\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168205    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-hostroot\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168230    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-host-run-netns\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168256    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-bin\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168264    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0e10c383-d5b0-4782-9b86-85e113ae6e1c-hostroot\") pod \"multus-h8rr2\" (UID: \"0e10c383-d5b0-4782-9b86-85e113ae6e1c\") " pod="openshift-multus/multus-h8rr2"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168281    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-modprobe-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168289    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-cni-bin\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.168304    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168305    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c14af69e-6fab-438d-82c2-4b0523bfba4a-host\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c14af69e-6fab-438d-82c2-4b0523bfba4a-host\") pod \"node-ca-nq2kt\" (UID: \"c14af69e-6fab-438d-82c2-4b0523bfba4a\") " pod="openshift-image-registry/node-ca-nq2kt"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.168353    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:40.668337665 +0000 UTC m=+1.985987667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : secret "metrics-daemon-secret" not found
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168374    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9441fd42-7d78-4065-8f63-462144f681a4-host-slash\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168387    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-etc-modprobe-d\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168394    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-netns\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168418    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-lib-modules\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168435    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9441fd42-7d78-4065-8f63-462144f681a4-host-slash\") pod \"iptables-alerter-nzcrt\" (UID: \"9441fd42-7d78-4065-8f63-462144f681a4\") " pod="openshift-network-operator/iptables-alerter-nzcrt"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168442    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-var-lib-kubelet\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.168414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168450    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/589a7de9-d92b-43e5-98eb-604e083bd543-host-run-netns\") pod \"ovnkube-node-bn45z\" (UID: \"589a7de9-d92b-43e5-98eb-604e083bd543\") " pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cnibin\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-var-lib-kubelet\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168516    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-lib-modules\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168522    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-sys\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168550    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/66738f34-062c-492c-9930-7f5882b3073d-sys\") pod \"tuned-wdllg\" (UID: \"66738f34-062c-492c-9930-7f5882b3073d\") " pod="openshift-cluster-node-tuning-operator/tuned-wdllg"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168556    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c-cnibin\") pod \"multus-additional-cni-plugins-q9hwj\" (UID: \"e0a25c3b-f254-46fb-a0dd-90f4c5cb4b9c\") " pod="openshift-multus/multus-additional-cni-plugins-q9hwj"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168586    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-socket-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168614    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-registration-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168629    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-socket-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.170561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.168654    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c93a44d2-477d-4501-b7ac-f681ba8f0ae9-registration-dir\") pod \"aws-ebs-csi-driver-node-mxdhw\" (UID: \"c93a44d2-477d-4501-b7ac-f681ba8f0ae9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mxdhw"
Nov 01 22:23:40.231292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.231275    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:23:40.673485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.673454    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:40.673627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.673501    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:40.673627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:40.673534    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:40.673627 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.673590    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:40.673627 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.673616    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:40.673755 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.673641    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:40.673755 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.673650    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.673633361 +0000 UTC m=+2.991283354 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:40.673755 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.673694    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.673682992 +0000 UTC m=+2.991332971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:40.673755 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:40.673705    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:41.673698928 +0000 UTC m=+2.991348906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : secret "metrics-daemon-secret" not found
Nov 01 22:23:41.565095 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:41.565069    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:41.679479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:41.679439    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:41.679640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:41.679521    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:41.679640 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:41.679567    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:41.679640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:41.679581    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:41.679640 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:41.679631    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:41.679851 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:41.679647    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.679627129 +0000 UTC m=+4.997277127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : secret "metrics-daemon-secret" not found
Nov 01 22:23:41.679851 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:41.679698    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.679679398 +0000 UTC m=+4.997329396 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:41.679851 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:41.679699    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:41.679851 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:41.679760    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:43.679748011 +0000 UTC m=+4.997398018 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:41.976431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:41.976411    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6hpf6_91c668ab-f0d3-45ee-a5c0-6f37c2ff8400/dns-node-resolver/0.log"
Nov 01 22:23:42.654297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:42.654255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:42.840219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:42.840182    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:43.089569 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.089535    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:43.376829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.376753    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nq2kt_c14af69e-6fab-438d-82c2-4b0523bfba4a/node-ca/0.log"
Nov 01 22:23:43.434132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.434099    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:23:43.694365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.694343    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.694391    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.694426    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:43.694485    4531 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:43.694533    4531 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:43.694548    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs podName:f9f3380f-6866-4ade-b5b8-17416d4bae89 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.694530788 +0000 UTC m=+9.012180787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs") pod "network-metrics-daemon-r9kzv" (UID: "f9f3380f-6866-4ade-b5b8-17416d4bae89") : secret "metrics-daemon-secret" not found
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:43.694566    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls podName:f1473442-fc77-41b5-8d1b-4c2d8aa380b3 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.694556548 +0000 UTC m=+9.012206539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls") pod "dns-default-gtxmw" (UID: "f1473442-fc77-41b5-8d1b-4c2d8aa380b3") : secret "dns-default-metrics-tls" not found
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:43.694570    4531 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:43.694675 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:43.694659    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls podName:b088fd75-2aab-4269-b9dd-4276185761ae nodeName:}" failed. No retries permitted until 2026-11-01 22:23:47.694641256 +0000 UTC m=+9.012291250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5qrr5" (UID: "b088fd75-2aab-4269-b9dd-4276185761ae") : secret "insights-runtime-extractor-tls" not found
Nov 01 22:23:43.857606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.857575    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-wph96" event={"ID":"4fcf42b1-1e74-4ccb-ae30-c0f893192c86","Type":"ContainerStarted","Data":"100119eb7e854ce9e7dc9bf6857decf3fabd666d5c869dfc3deda1c51ed08749"}
Nov 01 22:23:43.975837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:43.975816    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qqs5h_3b8ce285-174b-4405-80a1-4770b4706789/serve-healthcheck-canary/0.log"
Nov 01 22:23:44.576521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:44.576495    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-rmgcl_55686fc0-2a46-4674-bfc0-06754b4a02c9/migrator/0.log"
Nov 01 22:23:44.865807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:44.865742    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:45.269975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:45.269937    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:45.442415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:45.442384    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:45.803215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:45.803181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:46.008922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:46.008895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:46.199202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:46.199168    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:46.380822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:46.380796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:46.607171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:46.607118    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:47.391385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.391359    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:23:47.391810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.391545    4531 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 22:23:47.404134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.404103    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:47.720954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.720928    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:47.721100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.720978    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:47.721100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.721023    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:47.721336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.721319    4531 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 22:23:47.723366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.723343    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f1473442-fc77-41b5-8d1b-4c2d8aa380b3-metrics-tls\") pod \"dns-default-gtxmw\" (UID: \"f1473442-fc77-41b5-8d1b-4c2d8aa380b3\") " pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:47.723489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.723384    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b088fd75-2aab-4269-b9dd-4276185761ae-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5qrr5\" (UID: \"b088fd75-2aab-4269-b9dd-4276185761ae\") " pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:47.723489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.723384    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f9f3380f-6866-4ade-b5b8-17416d4bae89-metrics-certs\") pod \"network-metrics-daemon-r9kzv\" (UID: \"f9f3380f-6866-4ade-b5b8-17416d4bae89\") " pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:47.727316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.727293    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9kzv"
Nov 01 22:23:47.728416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.728400    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:47.728852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.728838    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5qrr5"
Nov 01 22:23:47.808259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:47.808172    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:48.006108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.006039    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:48.098182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.098143    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r9kzv"]
Nov 01 22:23:48.105868 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:48.105845    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9f3380f_6866_4ade_b5b8_17416d4bae89.slice/crio-c8b7a807a321418efc6d46826a04f92fd3c2cb2d6a0e3e5a7c9262d3bb859a7a WatchSource:0}: Error finding container c8b7a807a321418efc6d46826a04f92fd3c2cb2d6a0e3e5a7c9262d3bb859a7a: Status 404 returned error can't find the container with id c8b7a807a321418efc6d46826a04f92fd3c2cb2d6a0e3e5a7c9262d3bb859a7a
Nov 01 22:23:48.151391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.151365    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gtxmw"]
Nov 01 22:23:48.154663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.154642    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5qrr5"]
Nov 01 22:23:48.157867 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:48.157844    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1473442_fc77_41b5_8d1b_4c2d8aa380b3.slice/crio-68c9091ef45eeb4cab7d8ee060f38a1addcebfa8fd7bef3f95a8914b82489588 WatchSource:0}: Error finding container 68c9091ef45eeb4cab7d8ee060f38a1addcebfa8fd7bef3f95a8914b82489588: Status 404 returned error can't find the container with id 68c9091ef45eeb4cab7d8ee060f38a1addcebfa8fd7bef3f95a8914b82489588
Nov 01 22:23:48.161869 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:48.161842    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb088fd75_2aab_4269_b9dd_4276185761ae.slice/crio-e6a13275e6b4db52e69bbb7016154ec7182812505edc605ac511dbfa47c9bc31 WatchSource:0}: Error finding container e6a13275e6b4db52e69bbb7016154ec7182812505edc605ac511dbfa47c9bc31: Status 404 returned error can't find the container with id e6a13275e6b4db52e69bbb7016154ec7182812505edc605ac511dbfa47c9bc31
Nov 01 22:23:48.204975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.204952    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:48.607462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.607433    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:48.871281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.871213    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtxmw" event={"ID":"f1473442-fc77-41b5-8d1b-4c2d8aa380b3","Type":"ContainerStarted","Data":"68c9091ef45eeb4cab7d8ee060f38a1addcebfa8fd7bef3f95a8914b82489588"}
Nov 01 22:23:48.872300 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.872277    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r9kzv" event={"ID":"f9f3380f-6866-4ade-b5b8-17416d4bae89","Type":"ContainerStarted","Data":"c8b7a807a321418efc6d46826a04f92fd3c2cb2d6a0e3e5a7c9262d3bb859a7a"}
Nov 01 22:23:48.873537 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.873519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qrr5" event={"ID":"b088fd75-2aab-4269-b9dd-4276185761ae","Type":"ContainerStarted","Data":"09cc7f8f3b848c1b34589c33436b60bf020ceae3bee7f05ec2ecd70bb8f0cd0c"}
Nov 01 22:23:48.873603 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:48.873557    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qrr5" event={"ID":"b088fd75-2aab-4269-b9dd-4276185761ae","Type":"ContainerStarted","Data":"e6a13275e6b4db52e69bbb7016154ec7182812505edc605ac511dbfa47c9bc31"}
Nov 01 22:23:49.216607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:49.216579    4531 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-144.us-west-2.compute.internal" event="NodeReady"
Nov 01 22:23:49.411522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:49.411486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:49.623987 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:49.623955    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:49.879817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:49.879722    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qrr5" event={"ID":"b088fd75-2aab-4269-b9dd-4276185761ae","Type":"ContainerStarted","Data":"8c9133bd524c4d70c92211d279b72c493ce53149517b2d7b84cf54d657f6f1d6"}
Nov 01 22:23:50.013596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.013566    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:50.206334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.206303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:50.602850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.602826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:50.886161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.886083    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r9kzv" event={"ID":"f9f3380f-6866-4ade-b5b8-17416d4bae89","Type":"ContainerStarted","Data":"90d9fb8106e61bf1e36ea885c2603dc77f41c1c2e92a8d67cd36c47f77870ea1"}
Nov 01 22:23:50.886161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.886149    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r9kzv" event={"ID":"f9f3380f-6866-4ade-b5b8-17416d4bae89","Type":"ContainerStarted","Data":"26d57c005e69454b7feabfb38ec1f9d1577bd86acdf5c67adf6789ef973bed77"}
Nov 01 22:23:50.888485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.888452    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtxmw" event={"ID":"f1473442-fc77-41b5-8d1b-4c2d8aa380b3","Type":"ContainerStarted","Data":"6a9d168760dcada96f675b4166fd737f2e46034e3b878130d3acbd07d26d3909"}
Nov 01 22:23:50.888599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.888498    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gtxmw" event={"ID":"f1473442-fc77-41b5-8d1b-4c2d8aa380b3","Type":"ContainerStarted","Data":"9cdc7ba4a3d02893c16b51a4755149b6a4fe7250eb1db89f7108e3cde3bd3831"}
Nov 01 22:23:50.888599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:50.888586    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:23:51.206258 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:51.206225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:51.803350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:51.803310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:52.007142 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:52.007108    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:52.178769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:52.178743    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:52.406866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:52.406796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:52.582114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:52.582089    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:52.763271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:52.763243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:52.897003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:52.896979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5qrr5" event={"ID":"b088fd75-2aab-4269-b9dd-4276185761ae","Type":"ContainerStarted","Data":"387fbf337bd95266a5d55a7cb5e8ef9440010008cf70e782f6512a7624bebbff"}
Nov 01 22:23:53.145439 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:53.145375    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:53.316419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:53.316368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:53.486126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:53.486098    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:53.663898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:53.663874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:53.856617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:53.856554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:54.035096 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.035070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:54.404435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.404411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:54.613033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.613010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:54.800693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.800669    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-c9q7q"]
Nov 01 22:23:54.802759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.802719    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:54.847382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.847356    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"]
Nov 01 22:23:54.848985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.848967    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:54.857432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.857413    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 22:23:54.873684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.873660    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26ed46bd-1397-4ebb-83c0-3248be2542e5-tmp\") pod \"downloads-6dd789c8b7-c9q7q\" (UID: \"26ed46bd-1397-4ebb-83c0-3248be2542e5\") " pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:54.873783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.873688    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcc9j\" (UniqueName: \"kubernetes.io/projected/26ed46bd-1397-4ebb-83c0-3248be2542e5-kube-api-access-jcc9j\") pod \"downloads-6dd789c8b7-c9q7q\" (UID: \"26ed46bd-1397-4ebb-83c0-3248be2542e5\") " pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:54.873783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.873719    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4db06e67-53f1-49f8-8089-a13799623055-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv4xj\" (UID: \"4db06e67-53f1-49f8-8089-a13799623055\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:54.893379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.893362    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-jtnhf\""
Nov 01 22:23:54.893379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.893364    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 22:23:54.901866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.901848    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 22:23:54.902106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.902090    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-c57dh\""
Nov 01 22:23:54.902639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.902622    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c9q7q"]
Nov 01 22:23:54.917431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.917406    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"]
Nov 01 22:23:54.974024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.974001    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4db06e67-53f1-49f8-8089-a13799623055-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv4xj\" (UID: \"4db06e67-53f1-49f8-8089-a13799623055\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:54.974112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.974041    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26ed46bd-1397-4ebb-83c0-3248be2542e5-tmp\") pod \"downloads-6dd789c8b7-c9q7q\" (UID: \"26ed46bd-1397-4ebb-83c0-3248be2542e5\") " pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:54.974112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.974066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcc9j\" (UniqueName: \"kubernetes.io/projected/26ed46bd-1397-4ebb-83c0-3248be2542e5-kube-api-access-jcc9j\") pod \"downloads-6dd789c8b7-c9q7q\" (UID: \"26ed46bd-1397-4ebb-83c0-3248be2542e5\") " pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:54.974182 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:54.974140    4531 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Nov 01 22:23:54.974218 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:23:54.974195    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4db06e67-53f1-49f8-8089-a13799623055-tls-certificates podName:4db06e67-53f1-49f8-8089-a13799623055 nodeName:}" failed. No retries permitted until 2026-11-01 22:23:55.474180216 +0000 UTC m=+16.791830223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/4db06e67-53f1-49f8-8089-a13799623055-tls-certificates") pod "prometheus-operator-admission-webhook-6489d8bf89-cv4xj" (UID: "4db06e67-53f1-49f8-8089-a13799623055") : secret "prometheus-operator-admission-webhook-tls" not found
Nov 01 22:23:54.974429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:54.974412    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26ed46bd-1397-4ebb-83c0-3248be2542e5-tmp\") pod \"downloads-6dd789c8b7-c9q7q\" (UID: \"26ed46bd-1397-4ebb-83c0-3248be2542e5\") " pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:55.029849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.029827    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcc9j\" (UniqueName: \"kubernetes.io/projected/26ed46bd-1397-4ebb-83c0-3248be2542e5-kube-api-access-jcc9j\") pod \"downloads-6dd789c8b7-c9q7q\" (UID: \"26ed46bd-1397-4ebb-83c0-3248be2542e5\") " pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:55.082256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.082203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:55.113091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.113074    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:23:55.254248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.254214    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:55.446057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.446029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:55.478676 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.478653    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4db06e67-53f1-49f8-8089-a13799623055-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv4xj\" (UID: \"4db06e67-53f1-49f8-8089-a13799623055\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:55.480977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.480955    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/4db06e67-53f1-49f8-8089-a13799623055-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-cv4xj\" (UID: \"4db06e67-53f1-49f8-8089-a13799623055\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:55.523013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.522990    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-c9q7q"]
Nov 01 22:23:55.529105 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:55.529079    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ed46bd_1397_4ebb_83c0_3248be2542e5.slice/crio-c41b1a2e5f7d861bdae1a0d6572c4cf55865db24b14bf060a2c52dd150848b10 WatchSource:0}: Error finding container c41b1a2e5f7d861bdae1a0d6572c4cf55865db24b14bf060a2c52dd150848b10: Status 404 returned error can't find the container with id c41b1a2e5f7d861bdae1a0d6572c4cf55865db24b14bf060a2c52dd150848b10
Nov 01 22:23:55.634161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.634139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:55.757825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.757763    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:55.832637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.832613    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ggq5x"
Nov 01 22:23:55.855759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.855736    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:55.906768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:55.906743    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c9q7q" event={"ID":"26ed46bd-1397-4ebb-83c0-3248be2542e5","Type":"ContainerStarted","Data":"c41b1a2e5f7d861bdae1a0d6572c4cf55865db24b14bf060a2c52dd150848b10"}
Nov 01 22:23:56.030241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:56.030181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:56.116175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:56.116152    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"]
Nov 01 22:23:56.122848 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:23:56.122818    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db06e67_53f1_49f8_8089_a13799623055.slice/crio-9d942bfe503de9affaa151c50ca19c860fc7e1d2cc3d4c1cfe6e92c56ee9fca8 WatchSource:0}: Error finding container 9d942bfe503de9affaa151c50ca19c860fc7e1d2cc3d4c1cfe6e92c56ee9fca8: Status 404 returned error can't find the container with id 9d942bfe503de9affaa151c50ca19c860fc7e1d2cc3d4c1cfe6e92c56ee9fca8
Nov 01 22:23:56.219268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:56.219245    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:56.418169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:56.418113    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:56.910911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:56.910880    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj" event={"ID":"4db06e67-53f1-49f8-8089-a13799623055","Type":"ContainerStarted","Data":"9d942bfe503de9affaa151c50ca19c860fc7e1d2cc3d4c1cfe6e92c56ee9fca8"}
Nov 01 22:23:57.652684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:57.652635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:57.833946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:57.833918    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:57.915821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:57.915756    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj" event={"ID":"4db06e67-53f1-49f8-8089-a13799623055","Type":"ContainerStarted","Data":"656ce2aa2bf6d6ae368a7b956e4fcaec421e134873c6ccbb97b8ca67c0ace5b3"}
Nov 01 22:23:57.916199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:57.915967    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:57.920370 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:57.920352    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj"
Nov 01 22:23:58.023930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:58.023891    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-cv4xj" podStartSLOduration=2.426387997 podStartE2EDuration="4.023879286s" totalImagesPullingTime="1.597491289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:23:58.023519666 +0000 UTC m=+19.341169671" watchObservedRunningTime="2026-11-01 22:23:58.023879286 +0000 UTC m=+19.341529281"
Nov 01 22:23:58.489547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:58.489513    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:58.669722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:58.669692    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:58.861374 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:58.861297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:23:59.033618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:23:59.033586    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:00.894347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:00.894319    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gtxmw"
Nov 01 22:24:01.885765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.884602    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6477d76dd6-85rht"]
Nov 01 22:24:01.916481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.916455    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:01.943847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.943817    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6477d76dd6-85rht"]
Nov 01 22:24:01.989545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.989495    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 22:24:01.989545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.989495    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 22:24:01.989795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.989555    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 22:24:01.989869 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:01.989851    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 22:24:02.012532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.012507    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2nmsv\""
Nov 01 22:24:02.012665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.012510    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 22:24:02.016045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016018    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.016178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016054    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.016178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016133    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bd6p\" (UniqueName: \"kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.016280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016177    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.016280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016214    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.016280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.016394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.016290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117124    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117408 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117178    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117408 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117201    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117398    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bd6p\" (UniqueName: \"kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117443    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117475    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.117530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.117498    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.118071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.118045    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.118185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.118123    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.118185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.118177    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.118427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.118406    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.120072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.120050    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.120266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.120245    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.170857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.170800    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bd6p\" (UniqueName: \"kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p\") pod \"console-6477d76dd6-85rht\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") " pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.241822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.241788    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:02.612199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.612170    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6477d76dd6-85rht"]
Nov 01 22:24:02.618592 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:02.618562    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe86774d_d18c_4f5f_9726_8ef1930bb849.slice/crio-f989034d79cd2d8a6f5e1030fcfb1c0895270b6d178d5ca6e92a5c7c280ff9b4 WatchSource:0}: Error finding container f989034d79cd2d8a6f5e1030fcfb1c0895270b6d178d5ca6e92a5c7c280ff9b4: Status 404 returned error can't find the container with id f989034d79cd2d8a6f5e1030fcfb1c0895270b6d178d5ca6e92a5c7c280ff9b4
Nov 01 22:24:02.932980 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:02.932892    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477d76dd6-85rht" event={"ID":"be86774d-d18c-4f5f-9726-8ef1930bb849","Type":"ContainerStarted","Data":"f989034d79cd2d8a6f5e1030fcfb1c0895270b6d178d5ca6e92a5c7c280ff9b4"}
Nov 01 22:24:04.217743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:04.217701    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:04.816795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:04.816757    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:05.113326 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.113246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:05.290506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.290476    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:05.419380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.418840    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"]
Nov 01 22:24:05.422702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.422682    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.437646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.437623    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.437784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.437661    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.437784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.437687    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltdx\" (UniqueName: \"kubernetes.io/projected/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-kube-api-access-7ltdx\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.437902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.437809    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.471983 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.471954    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 01 22:24:05.477483 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.477414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:05.518216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.518166    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 22:24:05.518547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.518524    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2ltqz\""
Nov 01 22:24:05.520294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.518823    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 01 22:24:05.520294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.520052    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"]
Nov 01 22:24:05.520294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.520207    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 22:24:05.520768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.518772    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 22:24:05.538796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.538771    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.538941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.538812    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.538941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.538839    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltdx\" (UniqueName: \"kubernetes.io/projected/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-kube-api-access-7ltdx\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.538941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.538886    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.539626 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.539603    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.542141 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.542116    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.545448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.545424    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.567078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.567049    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-n97zh"]
Nov 01 22:24:05.570825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.570797    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"]
Nov 01 22:24:05.570974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.570948    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.573948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.573928    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.594292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.594267    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltdx\" (UniqueName: \"kubernetes.io/projected/11dc5494-1fd6-43c6-89d7-80fdd7b284bd-kube-api-access-7ltdx\") pod \"openshift-state-metrics-7bd7489957-cjzrk\" (UID: \"11dc5494-1fd6-43c6-89d7-80fdd7b284bd\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.638554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.638523    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 22:24:05.638685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.638584    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 22:24:05.638685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.638608    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 22:24:05.638848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.638759    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 22:24:05.638848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.638810    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pq899\""
Nov 01 22:24:05.638978 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.638875    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-8bwdj\""
Nov 01 22:24:05.639531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639507    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-tls\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.639638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639548    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509dedcc-8556-414c-b033-c1bb924bec68-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.639638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.639638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639585    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rcg\" (UniqueName: \"kubernetes.io/projected/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-kube-api-access-c9rcg\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.639638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639621    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-textfile\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.639884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639670    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.639884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzxtv\" (UniqueName: \"kubernetes.io/projected/509dedcc-8556-414c-b033-c1bb924bec68-kube-api-access-pzxtv\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.639884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639777    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-wtmp\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.639884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639818    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.639884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.639884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639867    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-sys\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.640186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.639946    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/509dedcc-8556-414c-b033-c1bb924bec68-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.640186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.640060    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-root\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.640186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.640093    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-metrics-client-ca\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.640186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.640125    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-accelerators-collector-config\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.665913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.665889    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"]
Nov 01 22:24:05.676220 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.676126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:05.700017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.699989    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 22:24:05.700133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.700115    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-accelerators-collector-config\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740652    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-tls\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740690    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509dedcc-8556-414c-b033-c1bb924bec68-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740720    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740763    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rcg\" (UniqueName: \"kubernetes.io/projected/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-kube-api-access-c9rcg\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740804    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-textfile\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740830    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740858    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzxtv\" (UniqueName: \"kubernetes.io/projected/509dedcc-8556-414c-b033-c1bb924bec68-kube-api-access-pzxtv\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740882    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-wtmp\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740921    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.740972    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-sys\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/509dedcc-8556-414c-b033-c1bb924bec68-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741087    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-root\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-root\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741194    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-accelerators-collector-config\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.741381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741211    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-textfile\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.741612    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-wtmp\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.742010    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-metrics-client-ca\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.742088    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/509dedcc-8556-414c-b033-c1bb924bec68-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.742125    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-sys\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.742371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/509dedcc-8556-414c-b033-c1bb924bec68-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.742564    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-metrics-client-ca\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.743419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.742779    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.744513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.744487    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-tls\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.744873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.744850    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.744958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.744873    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/509dedcc-8556-414c-b033-c1bb924bec68-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.746178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.746154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.795434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.795394    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rcg\" (UniqueName: \"kubernetes.io/projected/8d78fe3b-1aae-4121-9b0d-36d1eb1ece95-kube-api-access-c9rcg\") pod \"node-exporter-n97zh\" (UID: \"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95\") " pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.795434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.795415    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzxtv\" (UniqueName: \"kubernetes.io/projected/509dedcc-8556-414c-b033-c1bb924bec68-kube-api-access-pzxtv\") pod \"kube-state-metrics-5dc8d7d89b-7zwd6\" (UID: \"509dedcc-8556-414c-b033-c1bb924bec68\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.891403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.891202    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-n97zh"
Nov 01 22:24:05.898181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.898157    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"
Nov 01 22:24:05.906122 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:05.906094    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d78fe3b_1aae_4121_9b0d_36d1eb1ece95.slice/crio-2b9a260800a6ba1c0e1dd684c841e658c24c731287a84e5155bcaf6eed68909a WatchSource:0}: Error finding container 2b9a260800a6ba1c0e1dd684c841e658c24c731287a84e5155bcaf6eed68909a: Status 404 returned error can't find the container with id 2b9a260800a6ba1c0e1dd684c841e658c24c731287a84e5155bcaf6eed68909a
Nov 01 22:24:05.925163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.925112    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:05.945787 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:05.944068    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n97zh" event={"ID":"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95","Type":"ContainerStarted","Data":"2b9a260800a6ba1c0e1dd684c841e658c24c731287a84e5155bcaf6eed68909a"}
Nov 01 22:24:06.186535 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.186498    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk"]
Nov 01 22:24:06.193356 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:06.193323    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dc5494_1fd6_43c6_89d7_80fdd7b284bd.slice/crio-13b9ae043630544650a1e25fa526b9c4e0c74da07a774d288903e33bb028f637 WatchSource:0}: Error finding container 13b9ae043630544650a1e25fa526b9c4e0c74da07a774d288903e33bb028f637: Status 404 returned error can't find the container with id 13b9ae043630544650a1e25fa526b9c4e0c74da07a774d288903e33bb028f637
Nov 01 22:24:06.245822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.245789    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:06.290093 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.290067    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6"]
Nov 01 22:24:06.303812 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:06.303782    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod509dedcc_8556_414c_b033_c1bb924bec68.slice/crio-28db7b6075c9cbb84e5acae847848cfd22eb7e0d1d69a7a1aca54010ed3ef8ae WatchSource:0}: Error finding container 28db7b6075c9cbb84e5acae847848cfd22eb7e0d1d69a7a1aca54010ed3ef8ae: Status 404 returned error can't find the container with id 28db7b6075c9cbb84e5acae847848cfd22eb7e0d1d69a7a1aca54010ed3ef8ae
Nov 01 22:24:06.911301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.911248    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:06.951119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.951079    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk" event={"ID":"11dc5494-1fd6-43c6-89d7-80fdd7b284bd","Type":"ContainerStarted","Data":"fc5dc9593020e66f86305b80436732c7a56c690a42bc81be3d910085da35999b"}
Nov 01 22:24:06.951252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.951147    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk" event={"ID":"11dc5494-1fd6-43c6-89d7-80fdd7b284bd","Type":"ContainerStarted","Data":"3e4286bfa96e5d3a70c305e63439ce94e384db46ca5782d8481af1e5fa396183"}
Nov 01 22:24:06.951252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.951175    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk" event={"ID":"11dc5494-1fd6-43c6-89d7-80fdd7b284bd","Type":"ContainerStarted","Data":"13b9ae043630544650a1e25fa526b9c4e0c74da07a774d288903e33bb028f637"}
Nov 01 22:24:06.952892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.952865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6" event={"ID":"509dedcc-8556-414c-b033-c1bb924bec68","Type":"ContainerStarted","Data":"28db7b6075c9cbb84e5acae847848cfd22eb7e0d1d69a7a1aca54010ed3ef8ae"}
Nov 01 22:24:06.954525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:06.954500    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477d76dd6-85rht" event={"ID":"be86774d-d18c-4f5f-9726-8ef1930bb849","Type":"ContainerStarted","Data":"beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc"}
Nov 01 22:24:07.064812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.064760    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6477d76dd6-85rht" podStartSLOduration=2.395721817 podStartE2EDuration="6.064742698s" totalImagesPullingTime="3.669020881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:07.063342788 +0000 UTC m=+28.380992790" watchObservedRunningTime="2026-11-01 22:24:07.064742698 +0000 UTC m=+28.382392697"
Nov 01 22:24:07.268229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.268195    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:07.730076 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.728246    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:24:07.750280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.750256    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755002    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755068    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755109    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755130    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755149    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6hmp\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755179    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755219    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755249    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755309    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755380    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755414    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.755757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.755511    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:07.800073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.799573    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 22:24:07.800073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.799594    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 22:24:07.800073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.799625    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 22:24:07.800073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.799786    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 22:24:07.802778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.802760    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x9m5b\""
Nov 01 22:24:07.847755 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.847715    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:24:07.855949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.855917    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.855961    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.855994    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856026    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856094    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856119    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6hmp\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856151    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856216    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856250    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856283    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.856814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.856324    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.858505 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.857878    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.858505 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.858199    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.860200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.860176    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.860857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.860794    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.860857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.860816    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.862220 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.862185    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.862741 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.862572    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.893413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.893391    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 22:24:07.898195 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.898171    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.899943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.899800    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 22:24:07.899943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.899854    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 22:24:07.899943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.899886    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 22:24:07.911845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.911803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.914141 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.914120    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.915792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.915717    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.926053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.926032    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 22:24:07.932652 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.932610    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6hmp\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:07.934634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:07.934616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:08.155322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.155246    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"]
Nov 01 22:24:08.174364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.174329    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.185874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.185839    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:24:08.229134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.229051    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 22:24:08.230424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.229687    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 22:24:08.230424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.229766    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 22:24:08.230424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.229954    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 22:24:08.230424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.230173    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 22:24:08.230424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.230238    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-jncf7hf7s28o\""
Nov 01 22:24:08.230424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.230316    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6xctn\""
Nov 01 22:24:08.246517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.245543    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"]
Nov 01 22:24:08.362410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362377    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362429    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-tls\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6qj\" (UniqueName: \"kubernetes.io/projected/030ddc17-a369-45c0-9dc0-09c563cb22cc-kube-api-access-9b6qj\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362525    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362559    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362590    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030ddc17-a369-45c0-9dc0-09c563cb22cc-tmp\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362621    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/030ddc17-a369-45c0-9dc0-09c563cb22cc-metrics-client-ca\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362644    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-grpc-tls\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.362720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.362686    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-tls\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463665    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6qj\" (UniqueName: \"kubernetes.io/projected/030ddc17-a369-45c0-9dc0-09c563cb22cc-kube-api-access-9b6qj\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463692    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463720    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463768    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030ddc17-a369-45c0-9dc0-09c563cb22cc-tmp\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463799    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/030ddc17-a369-45c0-9dc0-09c563cb22cc-metrics-client-ca\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463820    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-grpc-tls\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.463860    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.464664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.464634    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/030ddc17-a369-45c0-9dc0-09c563cb22cc-tmp\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.464783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.464763    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/030ddc17-a369-45c0-9dc0-09c563cb22cc-metrics-client-ca\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.467405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.467359    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.468060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.468032    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.468717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.468690    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.469049 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.469025    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-tls\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.469536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.469512    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.469626 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.469537    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/030ddc17-a369-45c0-9dc0-09c563cb22cc-secret-grpc-tls\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.516861 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.516835    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6qj\" (UniqueName: \"kubernetes.io/projected/030ddc17-a369-45c0-9dc0-09c563cb22cc-kube-api-access-9b6qj\") pod \"thanos-querier-5f5c7bdb56-zp6cq\" (UID: \"030ddc17-a369-45c0-9dc0-09c563cb22cc\") " pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.568840 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.568818    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:08.642557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.642528    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:24:08.649371 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:08.649346    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9618be2d_cc7a_485d_9699_d15d49ce8614.slice/crio-79398e84da3c5abdec490abd19cb7ead888bb594c879a906b05bc05412889c60 WatchSource:0}: Error finding container 79398e84da3c5abdec490abd19cb7ead888bb594c879a906b05bc05412889c60: Status 404 returned error can't find the container with id 79398e84da3c5abdec490abd19cb7ead888bb594c879a906b05bc05412889c60
Nov 01 22:24:08.946185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.945004    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bn45z"
Nov 01 22:24:08.965198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.965169    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"79398e84da3c5abdec490abd19cb7ead888bb594c879a906b05bc05412889c60"}
Nov 01 22:24:08.966810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.966780    4531 generic.go:356] "Generic (PLEG): container finished" podID="8d78fe3b-1aae-4121-9b0d-36d1eb1ece95" containerID="8523f086591f85d1bdd3efbe9648713a149a980eb53e693cca7f6697d8f42817" exitCode=0
Nov 01 22:24:08.966913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.966853    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n97zh" event={"ID":"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95","Type":"ContainerDied","Data":"8523f086591f85d1bdd3efbe9648713a149a980eb53e693cca7f6697d8f42817"}
Nov 01 22:24:08.969358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:08.969335    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk" event={"ID":"11dc5494-1fd6-43c6-89d7-80fdd7b284bd","Type":"ContainerStarted","Data":"04d3ae31f623f48dabfa1454fdd4a83dfbf54385ff568abd67901dd7ad29efad"}
Nov 01 22:24:09.002384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.002361    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"]
Nov 01 22:24:09.009782 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:09.009759    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030ddc17_a369_45c0_9dc0_09c563cb22cc.slice/crio-bb814de11e06c1e99c7e476d5bffea6f1dc04f14b0d37390eab9b89f6bd6c8f5 WatchSource:0}: Error finding container bb814de11e06c1e99c7e476d5bffea6f1dc04f14b0d37390eab9b89f6bd6c8f5: Status 404 returned error can't find the container with id bb814de11e06c1e99c7e476d5bffea6f1dc04f14b0d37390eab9b89f6bd6c8f5
Nov 01 22:24:09.281697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.281554    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-cjzrk" podStartSLOduration=2.14064103 podStartE2EDuration="4.28153922s" totalImagesPullingTime="2.14089819s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:09.280585433 +0000 UTC m=+30.598235425" watchObservedRunningTime="2026-11-01 22:24:09.28153922 +0000 UTC m=+30.599189218"
Nov 01 22:24:09.610326 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.610251    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:09.814458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.814415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:09.979057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.979015    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6" event={"ID":"509dedcc-8556-414c-b033-c1bb924bec68","Type":"ContainerStarted","Data":"8001b7a3b49f500ba548ea7016ea94acd5b0233700fb021e7face6df547add86"}
Nov 01 22:24:09.979497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.979121    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6" event={"ID":"509dedcc-8556-414c-b033-c1bb924bec68","Type":"ContainerStarted","Data":"229de8863cbbeb1f826b8684f9a2405988148db46ae21a016dde766f43b00c68"}
Nov 01 22:24:09.979497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.979176    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6" event={"ID":"509dedcc-8556-414c-b033-c1bb924bec68","Type":"ContainerStarted","Data":"aa971b95255f76e2d2aaec5db17eee71b4426a5f3b91304e16153bff3e2a3679"}
Nov 01 22:24:09.981611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.981580    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"bb814de11e06c1e99c7e476d5bffea6f1dc04f14b0d37390eab9b89f6bd6c8f5"}
Nov 01 22:24:09.985543 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.985520    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n97zh" event={"ID":"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95","Type":"ContainerStarted","Data":"7e6dd9925fef59fd66d6ed7b11e7227e5ce36cd4f257f255183f16964c6bd7b8"}
Nov 01 22:24:09.985659 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:09.985568    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-n97zh" event={"ID":"8d78fe3b-1aae-4121-9b0d-36d1eb1ece95","Type":"ContainerStarted","Data":"8b8ece29ec2b41721d46fc7582908079287349f55f7fc0436d14d0eabd250f0e"}
Nov 01 22:24:10.028757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.028710    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:10.203835 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.203767    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-n97zh" podStartSLOduration=2.808577428 podStartE2EDuration="5.203748977s" totalImagesPullingTime="2.297854951s" totalInitContainerRuntime="97.316598ms" isStatefulPod=true podCreationTimestamp="2026-11-01 22:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:10.203168982 +0000 UTC m=+31.520819021" watchObservedRunningTime="2026-11-01 22:24:10.203748977 +0000 UTC m=+31.521398980"
Nov 01 22:24:10.204552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.204506    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7zwd6" podStartSLOduration=2.522970554 podStartE2EDuration="5.204495717s" totalImagesPullingTime="2.681525163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:10.092179112 +0000 UTC m=+31.409829113" watchObservedRunningTime="2026-11-01 22:24:10.204495717 +0000 UTC m=+31.522145718"
Nov 01 22:24:10.222708 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.222631    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:10.409545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.409468    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:10.597955 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.597924    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:10.991130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.991056    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef" exitCode=0
Nov 01 22:24:10.991505 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:10.991178    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"}
Nov 01 22:24:11.240304 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.240273    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-566c444b98-tzc48"]
Nov 01 22:24:11.263302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.263235    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287280    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-telemeter-client-tls\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287320    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-serving-certs-ca-bundle\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287433    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287480    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287551    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-metrics-client-ca\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287609    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-federate-client-tls\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287672    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-secret-telemeter-client\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.287762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.287745    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2kdf\" (UniqueName: \"kubernetes.io/projected/b3945423-2879-49c0-bee4-03f8e0902a9e-kube-api-access-r2kdf\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.302061 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.301976    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-566c444b98-tzc48"]
Nov 01 22:24:11.312704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.312677    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 22:24:11.312835 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.312683    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 22:24:11.312835 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.312788    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 22:24:11.312991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.312976    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 22:24:11.313048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.313021    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pd7kd\""
Nov 01 22:24:11.313048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.313000    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 22:24:11.359347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.359321    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 22:24:11.388267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388238    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-telemeter-client-tls\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388270    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-serving-certs-ca-bundle\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388309    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388327    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-metrics-client-ca\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388352    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-federate-client-tls\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388393    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-secret-telemeter-client\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.388554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.388431    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2kdf\" (UniqueName: \"kubernetes.io/projected/b3945423-2879-49c0-bee4-03f8e0902a9e-kube-api-access-r2kdf\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.389930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.389856    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-metrics-client-ca\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.390049 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.390007    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-serving-certs-ca-bundle\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.390221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.390197    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3945423-2879-49c0-bee4-03f8e0902a9e-telemeter-trusted-ca-bundle\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.391575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.391555    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-telemeter-client-tls\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.391916 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.391892    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.392014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.391992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-secret-telemeter-client\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.392540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.392520    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b3945423-2879-49c0-bee4-03f8e0902a9e-federate-client-tls\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.442244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.442218    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2kdf\" (UniqueName: \"kubernetes.io/projected/b3945423-2879-49c0-bee4-03f8e0902a9e-kube-api-access-r2kdf\") pod \"telemeter-client-566c444b98-tzc48\" (UID: \"b3945423-2879-49c0-bee4-03f8e0902a9e\") " pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.531120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.531055    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:11.617799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.617770    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48"
Nov 01 22:24:11.716344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:11.716316    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:12.132535 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.132501    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-566c444b98-tzc48"]
Nov 01 22:24:12.189818 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:12.189781    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3945423_2879_49c0_bee4_03f8e0902a9e.slice/crio-a13d96e4a8a7e76809e163adcc05564ad1902168737a59e342aee0e028d48df3 WatchSource:0}: Error finding container a13d96e4a8a7e76809e163adcc05564ad1902168737a59e342aee0e028d48df3: Status 404 returned error can't find the container with id a13d96e4a8a7e76809e163adcc05564ad1902168737a59e342aee0e028d48df3
Nov 01 22:24:12.242165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.242133    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:12.242324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.242220    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:12.249195 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.248958    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:12.262994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.262967    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b9584b78b-h8p46"]
Nov 01 22:24:12.282767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.282430    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.283016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.282959    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:12.295551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.295526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.295706    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxnx6\" (UniqueName: \"kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.295767    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.295824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.295913    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.295981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.296001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.296064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.296031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.328565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.328512    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9584b78b-h8p46"]
Nov 01 22:24:12.379270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.379236    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 01 22:24:12.397487 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397417    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397492    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397528    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxnx6\" (UniqueName: \"kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397556    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397600    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397642    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397679    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.397865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.397699    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.398309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.398278    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.398541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.398522    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.399052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.399029    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.399595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.399536    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.399715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.399659    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.400794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.400749    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.407277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.407245    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.455658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.455630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxnx6\" (UniqueName: \"kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6\") pod \"console-5b9584b78b-h8p46\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") " pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.464779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.464708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:12.534286 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:24:12.534252    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:24:12.609675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.609644    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:12.651293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.651182    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:12.930137 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:12.930098    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:13.002552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:13.001870    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48" event={"ID":"b3945423-2879-49c0-bee4-03f8e0902a9e","Type":"ContainerStarted","Data":"a13d96e4a8a7e76809e163adcc05564ad1902168737a59e342aee0e028d48df3"}
Nov 01 22:24:13.006644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:13.006614    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"dea8144172d4a46ffa4fed7def3875f3d8a57a20df9c22de9a8444280bb3ad71"}
Nov 01 22:24:13.006806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:13.006679    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"c08526df2b372d67085557c3ba2c6f4696b525e38507a4bb8af9ea47cf547d50"}
Nov 01 22:24:13.011379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:13.011358    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:13.026063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:13.026040    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9584b78b-h8p46"]
Nov 01 22:24:13.041070 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:13.034614    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4930803_240a_4d3a_8643_617a4da2917d.slice/crio-b69e1428ea6fa4b462d31129d80c2bb353e1a1a188b7790a32385996afcced58 WatchSource:0}: Error finding container b69e1428ea6fa4b462d31129d80c2bb353e1a1a188b7790a32385996afcced58: Status 404 returned error can't find the container with id b69e1428ea6fa4b462d31129d80c2bb353e1a1a188b7790a32385996afcced58
Nov 01 22:24:13.436367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:13.436338    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gw5f9"
Nov 01 22:24:14.013548 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.013513    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"b68330fbbfde46f165e92403911d360375b9cfbf2cfaaa9f8a24833a0be44461"}
Nov 01 22:24:14.015450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.015414    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9584b78b-h8p46" event={"ID":"d4930803-240a-4d3a-8643-617a4da2917d","Type":"ContainerStarted","Data":"2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0"}
Nov 01 22:24:14.015595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.015480    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9584b78b-h8p46" event={"ID":"d4930803-240a-4d3a-8643-617a4da2917d","Type":"ContainerStarted","Data":"b69e1428ea6fa4b462d31129d80c2bb353e1a1a188b7790a32385996afcced58"}
Nov 01 22:24:14.019037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.019009    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"}
Nov 01 22:24:14.019157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.019062    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"}
Nov 01 22:24:14.019157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.019087    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"}
Nov 01 22:24:14.019157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.019115    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"}
Nov 01 22:24:14.019157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.019138    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"}
Nov 01 22:24:14.129800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.128970    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b9584b78b-h8p46" podStartSLOduration=2.128955412 podStartE2EDuration="2.128955412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:14.128409213 +0000 UTC m=+35.446059214" watchObservedRunningTime="2026-11-01 22:24:14.128955412 +0000 UTC m=+35.446605406"
Nov 01 22:24:14.561922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:14.561889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:15.027345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.027312    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"871e069f0377f7c5fd08915e63b64b76f8df7aa68b3426d4555643470e149a3b"}
Nov 01 22:24:15.027450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.027378    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"15c91b0e275254cea2919ea429a17f938b90f5755c7cb8d1e266b1ce7f39d56a"}
Nov 01 22:24:15.027450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.027420    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" event={"ID":"030ddc17-a369-45c0-9dc0-09c563cb22cc","Type":"ContainerStarted","Data":"ab6f40163858d886343c439f0d07573c5792c84b3b8d8cfd74ae9f7afd83d162"}
Nov 01 22:24:15.027546 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.027462    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:15.031462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.031435    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerStarted","Data":"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"}
Nov 01 22:24:15.158642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.158588    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq" podStartSLOduration=1.882084716 podStartE2EDuration="7.1585701s" totalImagesPullingTime="5.276485384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:15.157052641 +0000 UTC m=+36.474702643" watchObservedRunningTime="2026-11-01 22:24:15.1585701 +0000 UTC m=+36.476220102"
Nov 01 22:24:15.273300 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:15.273217    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.904890098 podStartE2EDuration="8.273199908s" totalImagesPullingTime="5.34759613s" totalInitContainerRuntime="20.71368ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:15.272429935 +0000 UTC m=+36.590079939" watchObservedRunningTime="2026-11-01 22:24:15.273199908 +0000 UTC m=+36.590849909"
Nov 01 22:24:16.039116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:16.039068    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48" event={"ID":"b3945423-2879-49c0-bee4-03f8e0902a9e","Type":"ContainerStarted","Data":"27f8a96b30fd0c5d5af64b0110750c17c5fd3243f415c35b70bcc60dec5414b1"}
Nov 01 22:24:16.039588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:16.039146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48" event={"ID":"b3945423-2879-49c0-bee4-03f8e0902a9e","Type":"ContainerStarted","Data":"9af770b78aa410fd9eaf21c48970df6d14af74224e6aba61b329f251d3b69b23"}
Nov 01 22:24:16.039588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:16.039174    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48" event={"ID":"b3945423-2879-49c0-bee4-03f8e0902a9e","Type":"ContainerStarted","Data":"735dfb34028f9267804bb4aa630b3306b2c7eab2cf9592512c93004a67cc6a72"}
Nov 01 22:24:16.136872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:16.136845    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:16.154464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:16.154415    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-566c444b98-tzc48" podStartSLOduration=2.331386794 podStartE2EDuration="5.154400425s" totalImagesPullingTime="2.823013631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:16.152697641 +0000 UTC m=+37.470347665" watchObservedRunningTime="2026-11-01 22:24:16.154400425 +0000 UTC m=+37.472050427"
Nov 01 22:24:16.330020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:16.329950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:17.738963 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:17.738922    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:17.908684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:17.908650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:18.299401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:18.299370    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:18.480802 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:18.480776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:19.055692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:19.053707    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-c9q7q" event={"ID":"26ed46bd-1397-4ebb-83c0-3248be2542e5","Type":"ContainerStarted","Data":"1461c6391896b0447dfc710d66a3ff53bc08f53a227c75db12ee4e3ceb4830b8"}
Nov 01 22:24:19.055692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:19.053940    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:24:19.063677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:19.063645    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-c9q7q"
Nov 01 22:24:19.256357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:19.256289    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-c9q7q" podStartSLOduration=2.433770179 podStartE2EDuration="25.25626992s" totalImagesPullingTime="22.822499741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:19.208622895 +0000 UTC m=+40.526272901" watchObservedRunningTime="2026-11-01 22:24:19.25626992 +0000 UTC m=+40.573919922"
Nov 01 22:24:20.670737 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.670687    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9584b78b-h8p46"]
Nov 01 22:24:20.717257 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.717229    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-54c875b49-2srtf"]
Nov 01 22:24:20.745547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.745522    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.772405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.772311    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c875b49-2srtf"]
Nov 01 22:24:20.775938 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.775915    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776128    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776181    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776294    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776394    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvf7\" (UniqueName: \"kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.776503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.776468    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.877714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877679    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.877891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877749    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.877891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877797    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.877891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877817    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.877891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877855    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.877891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877878    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.878143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877904    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvf7\" (UniqueName: \"kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.878143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.877964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.878246 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.878178    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.879192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.878647    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.879192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.878652    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.879192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.879143    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.879393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.879283    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.881369 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.881342    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.881667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.881644    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:20.937651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:20.937599    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvf7\" (UniqueName: \"kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7\") pod \"console-54c875b49-2srtf\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") " pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:21.046005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:21.045978    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f5c7bdb56-zp6cq"
Nov 01 22:24:21.078073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:21.078032    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:21.480750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:21.480697    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54c875b49-2srtf"]
Nov 01 22:24:21.487636 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:21.487608    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35eaa797_aa6b_4128_889e_1ba55351ecda.slice/crio-40963261ff770b1c38c2d037778801d71a0e862a11e61adf554bdf63c4fa14b8 WatchSource:0}: Error finding container 40963261ff770b1c38c2d037778801d71a0e862a11e61adf554bdf63c4fa14b8: Status 404 returned error can't find the container with id 40963261ff770b1c38c2d037778801d71a0e862a11e61adf554bdf63c4fa14b8
Nov 01 22:24:22.068541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:22.068505    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c875b49-2srtf" event={"ID":"35eaa797-aa6b-4128-889e-1ba55351ecda","Type":"ContainerStarted","Data":"80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39"}
Nov 01 22:24:22.069043 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:22.068554    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c875b49-2srtf" event={"ID":"35eaa797-aa6b-4128-889e-1ba55351ecda","Type":"ContainerStarted","Data":"40963261ff770b1c38c2d037778801d71a0e862a11e61adf554bdf63c4fa14b8"}
Nov 01 22:24:22.184153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:22.184102    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-54c875b49-2srtf" podStartSLOduration=2.184085067 podStartE2EDuration="2.184085067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:22.182657193 +0000 UTC m=+43.500307190" watchObservedRunningTime="2026-11-01 22:24:22.184085067 +0000 UTC m=+43.501735070"
Nov 01 22:24:22.609922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:22.609884    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:27.240966 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:27.240934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:27.418188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:27.418159    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:27.612174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:27.612118    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:27.828854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:27.828829    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:28.059621 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:28.059595    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:31.079014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:31.078971    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:31.079589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:31.079070    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:31.083395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:31.083374    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:31.108046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:31.108022    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:24:31.325638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:31.325609    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6477d76dd6-85rht"]
Nov 01 22:24:33.185083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:33.185052    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:33.357365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:33.357338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:34.624737 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:34.624705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:34.802370 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:34.802347    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:42.525416 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:24:42.525377    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:24:45.694053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:45.694017    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b9584b78b-h8p46" podUID="d4930803-240a-4d3a-8643-617a4da2917d" containerName="console" containerID="cri-o://2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0" gracePeriod=15
Nov 01 22:24:45.959424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:45.959399    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:46.068188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068165    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068202    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068232    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068371    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068435    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6\" (UniqueName: \"kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068466    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068533    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068564    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config\") pod \"d4930803-240a-4d3a-8643-617a4da2917d\" (UID: \"d4930803-240a-4d3a-8643-617a4da2917d\") "
Nov 01 22:24:46.068615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068561    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:46.068842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068802    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:46.068842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068830    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-trusted-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.069044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068849    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:24:46.069044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.068860    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:46.069184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.069160    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:46.072083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.072056    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:46.072083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.072072    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "kube-api-access-hxnx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:46.072237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.072123    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config" pod "d4930803-240a-4d3a-8643-617a4da2917d" (UID: "d4930803-240a-4d3a-8643-617a4da2917d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:46.162333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.162308    4531 generic.go:356] "Generic (PLEG): container finished" podID="d4930803-240a-4d3a-8643-617a4da2917d" containerID="2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0" exitCode=0
Nov 01 22:24:46.162420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.162354    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9584b78b-h8p46"
Nov 01 22:24:46.162460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.162355    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9584b78b-h8p46" event={"ID":"d4930803-240a-4d3a-8643-617a4da2917d","Type":"ContainerDied","Data":"2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0"}
Nov 01 22:24:46.162532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.162512    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9584b78b-h8p46" event={"ID":"d4930803-240a-4d3a-8643-617a4da2917d","Type":"ContainerDied","Data":"b69e1428ea6fa4b462d31129d80c2bb353e1a1a188b7790a32385996afcced58"}
Nov 01 22:24:46.162566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.162550    4531 scope.go:122] "RemoveContainer" containerID="2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0"
Nov 01 22:24:46.169784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169761    4531 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4930803-240a-4d3a-8643-617a4da2917d-tmp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.169885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169787    4531 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-console-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.169885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169804    4531 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-oauth-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.169885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169818    4531 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d4930803-240a-4d3a-8643-617a4da2917d-console-serving-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.169885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169832    4531 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-oauth-serving-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.169885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169846    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxnx6\" (UniqueName: \"kubernetes.io/projected/d4930803-240a-4d3a-8643-617a4da2917d-kube-api-access-hxnx6\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.169885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.169860    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d4930803-240a-4d3a-8643-617a4da2917d-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:46.170942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.170926    4531 scope.go:122] "RemoveContainer" containerID="2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0"
Nov 01 22:24:46.171193 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:24:46.171173    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0\": container with ID starting with 2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0 not found: ID does not exist" containerID="2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0"
Nov 01 22:24:46.171262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.171205    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0"} err="failed to get container status \"2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0\": rpc error: code = NotFound desc = could not find container \"2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0\": container with ID starting with 2ef7f2b2a9080f42eda8db6ef764f520d629e58f5ef06db4f6c42494f37297e0 not found: ID does not exist"
Nov 01 22:24:46.314325 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.314266    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9584b78b-h8p46"]
Nov 01 22:24:46.316485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.316465    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b9584b78b-h8p46"]
Nov 01 22:24:46.813290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:46.813259    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4930803-240a-4d3a-8643-617a4da2917d" path="/var/lib/kubelet/pods/d4930803-240a-4d3a-8643-617a4da2917d/volumes"
Nov 01 22:24:50.327252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:50.327219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:50.500489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:50.500465    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:53.853466 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:53.853434    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2zsrp-nzjfx"]
Nov 01 22:24:53.853908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:53.853602    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx" podUID="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" containerName="insights-gathering" containerID="cri-o://fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72" gracePeriod=30
Nov 01 22:24:54.143806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.143780    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:24:54.192045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.192013    4531 generic.go:356] "Generic (PLEG): container finished" podID="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" containerID="fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72" exitCode=2
Nov 01 22:24:54.192165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.192055    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx"
Nov 01 22:24:54.192165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.192057    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx" event={"ID":"c87117bd-9b41-428a-8e18-f5af6fd2fd7f","Type":"ContainerDied","Data":"fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72"}
Nov 01 22:24:54.192165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.192111    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-2zsrp-nzjfx" event={"ID":"c87117bd-9b41-428a-8e18-f5af6fd2fd7f","Type":"ContainerDied","Data":"6ea2e98cd54edd844a3bec4942247f8315e3df9b38099aa379f84d6b063669d8"}
Nov 01 22:24:54.192165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.192135    4531 scope.go:122] "RemoveContainer" containerID="fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72"
Nov 01 22:24:54.199824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.199800    4531 scope.go:122] "RemoveContainer" containerID="fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72"
Nov 01 22:24:54.200071 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:24:54.200052    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72\": container with ID starting with fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72 not found: ID does not exist" containerID="fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72"
Nov 01 22:24:54.200122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.200082    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72"} err="failed to get container status \"fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72\": rpc error: code = NotFound desc = could not find container \"fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72\": container with ID starting with fb8d977426cdf3b034bae226426bce208a8acccd6671d2cfe0fc759f76508f72 not found: ID does not exist"
Nov 01 22:24:54.221239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221216    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") pod \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") "
Nov 01 22:24:54.221330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221258    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") pod \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") "
Nov 01 22:24:54.221330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") pod \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") "
Nov 01 22:24:54.221413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221334    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") pod \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\" (UID: \"c87117bd-9b41-428a-8e18-f5af6fd2fd7f\") "
Nov 01 22:24:54.221574 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221551    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path" pod "c87117bd-9b41-428a-8e18-f5af6fd2fd7f" (UID: "c87117bd-9b41-428a-8e18-f5af6fd2fd7f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:24:54.221641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221554    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle" pod "c87117bd-9b41-428a-8e18-f5af6fd2fd7f" (UID: "c87117bd-9b41-428a-8e18-f5af6fd2fd7f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:54.221641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.221609    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle" pod "c87117bd-9b41-428a-8e18-f5af6fd2fd7f" (UID: "c87117bd-9b41-428a-8e18-f5af6fd2fd7f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:54.223345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.223325    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk" pod "c87117bd-9b41-428a-8e18-f5af6fd2fd7f" (UID: "c87117bd-9b41-428a-8e18-f5af6fd2fd7f"). InnerVolumeSpecName "kube-api-access-zlnjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:54.254783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.254762    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-mpnn9-p62lt"]
Nov 01 22:24:54.255765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.255214    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" containerName="insights-gathering"
Nov 01 22:24:54.255765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.255237    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" containerName="insights-gathering"
Nov 01 22:24:54.255765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.255267    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d4930803-240a-4d3a-8643-617a4da2917d" containerName="console"
Nov 01 22:24:54.255765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.255276    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4930803-240a-4d3a-8643-617a4da2917d" containerName="console"
Nov 01 22:24:54.255765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.255367    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" containerName="insights-gathering"
Nov 01 22:24:54.255765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.255380    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4930803-240a-4d3a-8643-617a4da2917d" containerName="console"
Nov 01 22:24:54.285581 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.285556    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.310895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.310870    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mpnn9-p62lt"]
Nov 01 22:24:54.322803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.322781    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-trusted-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:54.322803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.322801    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-archives-path\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:54.322922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.322811    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-service-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:54.322922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.322820    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlnjk\" (UniqueName: \"kubernetes.io/projected/c87117bd-9b41-428a-8e18-f5af6fd2fd7f-kube-api-access-zlnjk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:54.424089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.424036    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.424089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.424088    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.424218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.424149    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.424218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.424193    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgcrj\" (UniqueName: \"kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.524996    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.525030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.525068    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgcrj\" (UniqueName: \"kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.525117    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525408 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.525388    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525703 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.525684    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.525703 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.525695    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.578304 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.578278    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgcrj\" (UniqueName: \"kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj\") pod \"periodic-gathering-mpnn9-p62lt\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") " pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.602313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.602294    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-2zsrp-nzjfx"]
Nov 01 22:24:54.606418 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.606398    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-2zsrp-nzjfx"]
Nov 01 22:24:54.608352 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.608338    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:24:54.813738 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.813699    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c87117bd-9b41-428a-8e18-f5af6fd2fd7f" path="/var/lib/kubelet/pods/c87117bd-9b41-428a-8e18-f5af6fd2fd7f/volumes"
Nov 01 22:24:54.994129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:54.994102    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-mpnn9-p62lt"]
Nov 01 22:24:55.000600 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:24:55.000578    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89590d48_5c72_4c96_a46f_9d3a2a809810.slice/crio-82c3617aa90afdcf6efe05567c903ede2dd28a5f16d0432098267441562ca90b WatchSource:0}: Error finding container 82c3617aa90afdcf6efe05567c903ede2dd28a5f16d0432098267441562ca90b: Status 404 returned error can't find the container with id 82c3617aa90afdcf6efe05567c903ede2dd28a5f16d0432098267441562ca90b
Nov 01 22:24:55.197412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:55.197371    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mpnn9-p62lt" event={"ID":"89590d48-5c72-4c96-a46f-9d3a2a809810","Type":"ContainerStarted","Data":"ba72debe6aed0f9c1e1c7078ee83fef1e699d99fe606ffd7728239b29d01e274"}
Nov 01 22:24:55.197561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:55.197428    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mpnn9-p62lt" event={"ID":"89590d48-5c72-4c96-a46f-9d3a2a809810","Type":"ContainerStarted","Data":"82c3617aa90afdcf6efe05567c903ede2dd28a5f16d0432098267441562ca90b"}
Nov 01 22:24:55.304364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:55.304288    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-mpnn9-p62lt" podStartSLOduration=1.304275691 podStartE2EDuration="1.304275691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:24:55.303132322 +0000 UTC m=+76.620782321" watchObservedRunningTime="2026-11-01 22:24:55.304275691 +0000 UTC m=+76.621925686"
Nov 01 22:24:56.344623 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.344566    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6477d76dd6-85rht" podUID="be86774d-d18c-4f5f-9726-8ef1930bb849" containerName="console" containerID="cri-o://beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc" gracePeriod=15
Nov 01 22:24:56.619267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.619214    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:56.745553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745530    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.745675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745588    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.745675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745616    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.745675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745636    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.745675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745666    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.745912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745698    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p\" (UniqueName: \"kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.745912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.745759    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp\" (UniqueName: \"kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp\") pod \"be86774d-d18c-4f5f-9726-8ef1930bb849\" (UID: \"be86774d-d18c-4f5f-9726-8ef1930bb849\") "
Nov 01 22:24:56.746092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.746060    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:56.746250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.746082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:56.746250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.746129    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:24:56.746250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.746182    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:24:56.747809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.747783    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "kube-api-access-8bd6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:24:56.747893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.747814    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:56.747893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.747841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert" pod "be86774d-d18c-4f5f-9726-8ef1930bb849" (UID: "be86774d-d18c-4f5f-9726-8ef1930bb849"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:24:56.847090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847066    4531 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-oauth-serving-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:56.847090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847087    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:56.847215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847097    4531 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-oauth-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:56.847215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847106    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bd6p\" (UniqueName: \"kubernetes.io/projected/be86774d-d18c-4f5f-9726-8ef1930bb849-kube-api-access-8bd6p\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:56.847215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847116    4531 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/be86774d-d18c-4f5f-9726-8ef1930bb849-tmp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:56.847215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847125    4531 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/be86774d-d18c-4f5f-9726-8ef1930bb849-console-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:56.847215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:56.847133    4531 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/be86774d-d18c-4f5f-9726-8ef1930bb849-console-serving-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:24:57.205996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.205969    4531 generic.go:356] "Generic (PLEG): container finished" podID="be86774d-d18c-4f5f-9726-8ef1930bb849" containerID="beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc" exitCode=0
Nov 01 22:24:57.206123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.206001    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477d76dd6-85rht" event={"ID":"be86774d-d18c-4f5f-9726-8ef1930bb849","Type":"ContainerDied","Data":"beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc"}
Nov 01 22:24:57.206123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.206013    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477d76dd6-85rht"
Nov 01 22:24:57.206123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.206034    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477d76dd6-85rht" event={"ID":"be86774d-d18c-4f5f-9726-8ef1930bb849","Type":"ContainerDied","Data":"f989034d79cd2d8a6f5e1030fcfb1c0895270b6d178d5ca6e92a5c7c280ff9b4"}
Nov 01 22:24:57.206123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.206053    4531 scope.go:122] "RemoveContainer" containerID="beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc"
Nov 01 22:24:57.213831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.213815    4531 scope.go:122] "RemoveContainer" containerID="beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc"
Nov 01 22:24:57.214083 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:24:57.214062    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc\": container with ID starting with beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc not found: ID does not exist" containerID="beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc"
Nov 01 22:24:57.214137 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.214091    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc"} err="failed to get container status \"beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc\": rpc error: code = NotFound desc = could not find container \"beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc\": container with ID starting with beef186f224ef2ccabc305acc553aa96857d81ff45566f7ceec1fe8a4642ccdc not found: ID does not exist"
Nov 01 22:24:57.358516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.358493    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6477d76dd6-85rht"]
Nov 01 22:24:57.405135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.405111    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6477d76dd6-85rht"]
Nov 01 22:24:57.647488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.647424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:57.823469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:57.823439    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:24:58.813749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:24:58.813702    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be86774d-d18c-4f5f-9726-8ef1930bb849" path="/var/lib/kubelet/pods/be86774d-d18c-4f5f-9726-8ef1930bb849/volumes"
Nov 01 22:25:04.272358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:04.272324    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:05.056750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:05.056705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:08.237366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.237335    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b566d54b4-nwk7g"]
Nov 01 22:25:08.237751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.237630    4531 cpu_manager.go:414] "containerMap: removing container" podUID="be86774d-d18c-4f5f-9726-8ef1930bb849" containerName="console"
Nov 01 22:25:08.237751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.237641    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be86774d-d18c-4f5f-9726-8ef1930bb849" containerName="console"
Nov 01 22:25:08.237751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.237714    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="be86774d-d18c-4f5f-9726-8ef1930bb849" containerName="console"
Nov 01 22:25:08.240787 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.240768    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331590 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331569    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-serving-cert\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331618    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-trusted-ca-bundle\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331636    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-oauth-serving-cert\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331828 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331768    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-config\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331828 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331793    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-oauth-config\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331828 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331810    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrlz6\" (UniqueName: \"kubernetes.io/projected/acd375a7-1600-4402-86a2-5c1b7c0f86d1-kube-api-access-qrlz6\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331834    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd375a7-1600-4402-86a2-5c1b7c0f86d1-tmp\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.331929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.331880    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-service-ca\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.335723 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.335702    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b566d54b4-nwk7g"]
Nov 01 22:25:08.432764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.432735    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-serving-cert\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.432871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.432778    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-trusted-ca-bundle\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.432871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.432801    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-oauth-serving-cert\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.432871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.432828    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-config\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.432871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.432861    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-oauth-config\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.432896    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrlz6\" (UniqueName: \"kubernetes.io/projected/acd375a7-1600-4402-86a2-5c1b7c0f86d1-kube-api-access-qrlz6\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433079    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd375a7-1600-4402-86a2-5c1b7c0f86d1-tmp\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433115    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-service-ca\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433461    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/acd375a7-1600-4402-86a2-5c1b7c0f86d1-tmp\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-config\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433644    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-oauth-serving-cert\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433678    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-trusted-ca-bundle\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.433778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.433768    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/acd375a7-1600-4402-86a2-5c1b7c0f86d1-service-ca\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.435326 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.435308    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-oauth-config\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.435615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.435593    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/acd375a7-1600-4402-86a2-5c1b7c0f86d1-console-serving-cert\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.486202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.486179    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrlz6\" (UniqueName: \"kubernetes.io/projected/acd375a7-1600-4402-86a2-5c1b7c0f86d1-kube-api-access-qrlz6\") pod \"console-5b566d54b4-nwk7g\" (UID: \"acd375a7-1600-4402-86a2-5c1b7c0f86d1\") " pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.551354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.551305    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:08.571892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.571868    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:08.744903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.744874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:08.916139 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.916084    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b566d54b4-nwk7g"]
Nov 01 22:25:08.923560 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:25:08.923531    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacd375a7_1600_4402_86a2_5c1b7c0f86d1.slice/crio-928b86b6f836e6f8370e56d1b2cafc63116f73dfaaf278f9e3ed71afc74edc35 WatchSource:0}: Error finding container 928b86b6f836e6f8370e56d1b2cafc63116f73dfaaf278f9e3ed71afc74edc35: Status 404 returned error can't find the container with id 928b86b6f836e6f8370e56d1b2cafc63116f73dfaaf278f9e3ed71afc74edc35
Nov 01 22:25:08.946522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:08.946494    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:09.152188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:09.152157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:09.253764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:09.253714    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b566d54b4-nwk7g" event={"ID":"acd375a7-1600-4402-86a2-5c1b7c0f86d1","Type":"ContainerStarted","Data":"c0ef8437cae1a2b0608044d16b7b6be9932bcac4cd27c83fc54bbc255b95a86b"}
Nov 01 22:25:09.254121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:09.253824    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b566d54b4-nwk7g" event={"ID":"acd375a7-1600-4402-86a2-5c1b7c0f86d1","Type":"ContainerStarted","Data":"928b86b6f836e6f8370e56d1b2cafc63116f73dfaaf278f9e3ed71afc74edc35"}
Nov 01 22:25:09.363788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:09.363719    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b566d54b4-nwk7g" podStartSLOduration=1.363706688 podStartE2EDuration="1.363706688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:09.36308338 +0000 UTC m=+90.680733378" watchObservedRunningTime="2026-11-01 22:25:09.363706688 +0000 UTC m=+90.681356690"
Nov 01 22:25:11.821315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821282    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:25:11.821788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821571    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="alertmanager" containerID="cri-o://f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454" gracePeriod=120
Nov 01 22:25:11.821788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821693    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="config-reloader" containerID="cri-o://d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d" gracePeriod=120
Nov 01 22:25:11.821926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821826    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-metric" containerID="cri-o://8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2" gracePeriod=120
Nov 01 22:25:11.821926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821855    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="prom-label-proxy" containerID="cri-o://fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7" gracePeriod=120
Nov 01 22:25:11.821926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821863    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-web" containerID="cri-o://dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783" gracePeriod=120
Nov 01 22:25:11.822080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:11.821988    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy" containerID="cri-o://97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced" gracePeriod=120
Nov 01 22:25:12.269335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269303    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7" exitCode=0
Nov 01 22:25:12.269335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269330    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced" exitCode=0
Nov 01 22:25:12.269597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269369    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d" exitCode=0
Nov 01 22:25:12.269597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269372    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"}
Nov 01 22:25:12.269597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269431    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"}
Nov 01 22:25:12.269597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269381    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454" exitCode=0
Nov 01 22:25:12.269597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269471    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"}
Nov 01 22:25:12.269597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:12.269498    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"}
Nov 01 22:25:12.523322 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:12.523257    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:25:13.073278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.073255    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.170854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.170799    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.170854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.170838    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.170877    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.170904    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.170945    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171021    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171068    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171145    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171184    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171218    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171248    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171251    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:13.171471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171279    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db\") pod \"9618be2d-cc7a-485d-9699-d15d49ce8614\" (UID: \"9618be2d-cc7a-485d-9699-d15d49ce8614\") "
Nov 01 22:25:13.171627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171508    4531 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.171843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.171822    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:25:13.172380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.172350    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:13.173637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.173612    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:25:13.174082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.174047    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:25:13.174082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.174057    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.174664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.174625    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.174775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.174707    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.174775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.174764    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.176031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.176010    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "kube-api-access-b6hmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:25:13.176128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.176051    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.178339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.178251    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.185631 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.185611    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config" pod "9618be2d-cc7a-485d-9699-d15d49ce8614" (UID: "9618be2d-cc7a-485d-9699-d15d49ce8614"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:13.271782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271765    4531 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-web-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271783    4531 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-cluster-tls-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271793    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6hmp\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-kube-api-access-b6hmp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271803    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271812    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271821    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271830    4531 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271839    4531 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-alertmanager-main-db\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271847    4531 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9618be2d-cc7a-485d-9699-d15d49ce8614-tls-assets\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271855    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9618be2d-cc7a-485d-9699-d15d49ce8614-metrics-client-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271863    4531 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9618be2d-cc7a-485d-9699-d15d49ce8614-config-out\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.271870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.271872    4531 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/9618be2d-cc7a-485d-9699-d15d49ce8614-secret-alertmanager-main-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:13.276228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276205    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2" exitCode=0
Nov 01 22:25:13.276228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276224    4531 generic.go:356] "Generic (PLEG): container finished" podID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerID="dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783" exitCode=0
Nov 01 22:25:13.276361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276274    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.276361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276284    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"}
Nov 01 22:25:13.276435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276372    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"}
Nov 01 22:25:13.276435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276402    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"9618be2d-cc7a-485d-9699-d15d49ce8614","Type":"ContainerDied","Data":"79398e84da3c5abdec490abd19cb7ead888bb594c879a906b05bc05412889c60"}
Nov 01 22:25:13.276435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.276431    4531 scope.go:122] "RemoveContainer" containerID="fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"
Nov 01 22:25:13.283958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.283942    4531 scope.go:122] "RemoveContainer" containerID="8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"
Nov 01 22:25:13.290753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.290715    4531 scope.go:122] "RemoveContainer" containerID="97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"
Nov 01 22:25:13.297084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.297066    4531 scope.go:122] "RemoveContainer" containerID="dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"
Nov 01 22:25:13.303236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.303222    4531 scope.go:122] "RemoveContainer" containerID="d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"
Nov 01 22:25:13.309712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.309692    4531 scope.go:122] "RemoveContainer" containerID="f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"
Nov 01 22:25:13.316301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.316286    4531 scope.go:122] "RemoveContainer" containerID="2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"
Nov 01 22:25:13.322596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.322581    4531 scope.go:122] "RemoveContainer" containerID="fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"
Nov 01 22:25:13.322868 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.322850    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7\": container with ID starting with fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7 not found: ID does not exist" containerID="fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"
Nov 01 22:25:13.322930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.322876    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"} err="failed to get container status \"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7\": rpc error: code = NotFound desc = could not find container \"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7\": container with ID starting with fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7 not found: ID does not exist"
Nov 01 22:25:13.322930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.322896    4531 scope.go:122] "RemoveContainer" containerID="8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"
Nov 01 22:25:13.323128 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.323109    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2\": container with ID starting with 8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2 not found: ID does not exist" containerID="8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"
Nov 01 22:25:13.323196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323137    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"} err="failed to get container status \"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2\": rpc error: code = NotFound desc = could not find container \"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2\": container with ID starting with 8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2 not found: ID does not exist"
Nov 01 22:25:13.323196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323160    4531 scope.go:122] "RemoveContainer" containerID="97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"
Nov 01 22:25:13.323390 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.323373    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced\": container with ID starting with 97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced not found: ID does not exist" containerID="97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"
Nov 01 22:25:13.323427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323395    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"} err="failed to get container status \"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced\": rpc error: code = NotFound desc = could not find container \"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced\": container with ID starting with 97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced not found: ID does not exist"
Nov 01 22:25:13.323427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323409    4531 scope.go:122] "RemoveContainer" containerID="dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"
Nov 01 22:25:13.323609 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.323592    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783\": container with ID starting with dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783 not found: ID does not exist" containerID="dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"
Nov 01 22:25:13.323670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323615    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"} err="failed to get container status \"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783\": rpc error: code = NotFound desc = could not find container \"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783\": container with ID starting with dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783 not found: ID does not exist"
Nov 01 22:25:13.323670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323634    4531 scope.go:122] "RemoveContainer" containerID="d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"
Nov 01 22:25:13.323870 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.323855    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d\": container with ID starting with d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d not found: ID does not exist" containerID="d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"
Nov 01 22:25:13.323927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323874    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"} err="failed to get container status \"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d\": rpc error: code = NotFound desc = could not find container \"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d\": container with ID starting with d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d not found: ID does not exist"
Nov 01 22:25:13.323927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.323888    4531 scope.go:122] "RemoveContainer" containerID="f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"
Nov 01 22:25:13.324098 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.324080    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454\": container with ID starting with f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454 not found: ID does not exist" containerID="f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"
Nov 01 22:25:13.324162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324105    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"} err="failed to get container status \"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454\": rpc error: code = NotFound desc = could not find container \"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454\": container with ID starting with f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454 not found: ID does not exist"
Nov 01 22:25:13.324162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324123    4531 scope.go:122] "RemoveContainer" containerID="2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"
Nov 01 22:25:13.324336 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:13.324319    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef\": container with ID starting with 2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef not found: ID does not exist" containerID="2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"
Nov 01 22:25:13.324375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324341    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"} err="failed to get container status \"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef\": rpc error: code = NotFound desc = could not find container \"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef\": container with ID starting with 2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef not found: ID does not exist"
Nov 01 22:25:13.324375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324354    4531 scope.go:122] "RemoveContainer" containerID="fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"
Nov 01 22:25:13.324576 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324558    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7"} err="failed to get container status \"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7\": rpc error: code = NotFound desc = could not find container \"fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7\": container with ID starting with fd756d1c347b2ef6df9e28b896e87bcc4a3c83c3e880828c976c8293c986b2e7 not found: ID does not exist"
Nov 01 22:25:13.324642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324577    4531 scope.go:122] "RemoveContainer" containerID="8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"
Nov 01 22:25:13.324793 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324777    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2"} err="failed to get container status \"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2\": rpc error: code = NotFound desc = could not find container \"8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2\": container with ID starting with 8482cd8dfdc5247dc02230a98cdbb2c99fb2c91f75a3165a50660a81f9b5f9b2 not found: ID does not exist"
Nov 01 22:25:13.324843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324793    4531 scope.go:122] "RemoveContainer" containerID="97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"
Nov 01 22:25:13.324998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324980    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced"} err="failed to get container status \"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced\": rpc error: code = NotFound desc = could not find container \"97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced\": container with ID starting with 97acbeff3285cb6018bc5e240a268d384aae5da05055550359b8007a07a9fced not found: ID does not exist"
Nov 01 22:25:13.325064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.324999    4531 scope.go:122] "RemoveContainer" containerID="dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"
Nov 01 22:25:13.325200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325185    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783"} err="failed to get container status \"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783\": rpc error: code = NotFound desc = could not find container \"dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783\": container with ID starting with dfb87b02b616b64fd804a939a3b16ac24c7d3d44b217d0541015d829b07e6783 not found: ID does not exist"
Nov 01 22:25:13.325245 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325201    4531 scope.go:122] "RemoveContainer" containerID="d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"
Nov 01 22:25:13.325392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325374    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d"} err="failed to get container status \"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d\": rpc error: code = NotFound desc = could not find container \"d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d\": container with ID starting with d21a60412e261f44cd3b89043fc0abd71bde7151abe7dc50b4e24964106ce11d not found: ID does not exist"
Nov 01 22:25:13.325458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325392    4531 scope.go:122] "RemoveContainer" containerID="f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"
Nov 01 22:25:13.325586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325571    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454"} err="failed to get container status \"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454\": rpc error: code = NotFound desc = could not find container \"f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454\": container with ID starting with f9adafd510383e7cbf1c0c8baebbd168da361de83403fdb8cb3fd53a62c19454 not found: ID does not exist"
Nov 01 22:25:13.325629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325585    4531 scope.go:122] "RemoveContainer" containerID="2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"
Nov 01 22:25:13.325784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.325766    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef"} err="failed to get container status \"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef\": rpc error: code = NotFound desc = could not find container \"2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef\": container with ID starting with 2395f8c814110fdc891ab2c169de6e462fc2e0ca387a7d5ba85d0fcc41f62eef not found: ID does not exist"
Nov 01 22:25:13.436184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.436122    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:25:13.439010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.438979    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:25:13.464675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464652    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:25:13.464948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464936    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="alertmanager"
Nov 01 22:25:13.464993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464950    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="alertmanager"
Nov 01 22:25:13.464993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464963    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="prom-label-proxy"
Nov 01 22:25:13.464993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464968    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="prom-label-proxy"
Nov 01 22:25:13.464993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464977    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="init-config-reloader"
Nov 01 22:25:13.464993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464982    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="init-config-reloader"
Nov 01 22:25:13.464993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464993    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-metric"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.464998    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-metric"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465009    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="config-reloader"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465014    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="config-reloader"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465028    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-web"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465032    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-web"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465038    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465042    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465088    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-web"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465097    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465103    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="alertmanager"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465110    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="prom-label-proxy"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465117    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="kube-rbac-proxy-metric"
Nov 01 22:25:13.465164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.465123    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" containerName="config-reloader"
Nov 01 22:25:13.468715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.468700    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.518056 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518034    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 22:25:13.518248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518228    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 22:25:13.518341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518270    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 22:25:13.518341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518282    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 22:25:13.518341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518272    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 22:25:13.518341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518308    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 22:25:13.518341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518328    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 22:25:13.518613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518600    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-x9m5b\""
Nov 01 22:25:13.518723 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518705    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 22:25:13.518887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.518872    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 22:25:13.565564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.565544    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:25:13.573217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573199    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3075de8-ae16-4b2d-9199-d31c4726025f-config-out\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573228    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3075de8-ae16-4b2d-9199-d31c4726025f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573254    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573271    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573329    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573445 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573428    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573483 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573469    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573517    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3075de8-ae16-4b2d-9199-d31c4726025f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3075de8-ae16-4b2d-9199-d31c4726025f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wc2c\" (UniqueName: \"kubernetes.io/projected/c3075de8-ae16-4b2d-9199-d31c4726025f-kube-api-access-4wc2c\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573596    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573673    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3075de8-ae16-4b2d-9199-d31c4726025f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.573744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.573696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-web-config\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.674809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.674787    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.674915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.674821    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3075de8-ae16-4b2d-9199-d31c4726025f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.674915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.674841    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-web-config\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.674915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.674882    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3075de8-ae16-4b2d-9199-d31c4726025f-config-out\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.674941    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3075de8-ae16-4b2d-9199-d31c4726025f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.674989    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675027    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675052    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675106    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675164    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3075de8-ae16-4b2d-9199-d31c4726025f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3075de8-ae16-4b2d-9199-d31c4726025f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675217    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wc2c\" (UniqueName: \"kubernetes.io/projected/c3075de8-ae16-4b2d-9199-d31c4726025f-kube-api-access-4wc2c\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.675468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.675259    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c3075de8-ae16-4b2d-9199-d31c4726025f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.676337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.676312    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c3075de8-ae16-4b2d-9199-d31c4726025f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.676642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.676617    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c3075de8-ae16-4b2d-9199-d31c4726025f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.677986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.677961    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c3075de8-ae16-4b2d-9199-d31c4726025f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.678087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.677985    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.678216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.678193    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.678405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.678384    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c3075de8-ae16-4b2d-9199-d31c4726025f-config-out\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.678488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.678469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.678556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.678510    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.678663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.678646    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.679198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.679179    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-web-config\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.679290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.679208    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c3075de8-ae16-4b2d-9199-d31c4726025f-config-volume\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.728063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.728041    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wc2c\" (UniqueName: \"kubernetes.io/projected/c3075de8-ae16-4b2d-9199-d31c4726025f-kube-api-access-4wc2c\") pod \"alertmanager-main-0\" (UID: \"c3075de8-ae16-4b2d-9199-d31c4726025f\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:13.785950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:13.785929    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 22:25:14.205820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:14.205791    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 22:25:14.212334 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:25:14.212308    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3075de8_ae16_4b2d_9199_d31c4726025f.slice/crio-771dab85e31f3a87365e86b2234dc788dd2b662e90cc2026b1ed49b45b39b66a WatchSource:0}: Error finding container 771dab85e31f3a87365e86b2234dc788dd2b662e90cc2026b1ed49b45b39b66a: Status 404 returned error can't find the container with id 771dab85e31f3a87365e86b2234dc788dd2b662e90cc2026b1ed49b45b39b66a
Nov 01 22:25:14.284179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:14.284147    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"b30449bdc5f9d91110a419136e5b65d124c16c62943c9200cf146d0ce50064b6"}
Nov 01 22:25:14.284319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:14.284215    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"771dab85e31f3a87365e86b2234dc788dd2b662e90cc2026b1ed49b45b39b66a"}
Nov 01 22:25:14.817547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:14.817517    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9618be2d-cc7a-485d-9699-d15d49ce8614" path="/var/lib/kubelet/pods/9618be2d-cc7a-485d-9699-d15d49ce8614/volumes"
Nov 01 22:25:15.289978 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:15.289945    4531 generic.go:356] "Generic (PLEG): container finished" podID="c3075de8-ae16-4b2d-9199-d31c4726025f" containerID="b30449bdc5f9d91110a419136e5b65d124c16c62943c9200cf146d0ce50064b6" exitCode=0
Nov 01 22:25:15.290432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:15.290029    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerDied","Data":"b30449bdc5f9d91110a419136e5b65d124c16c62943c9200cf146d0ce50064b6"}
Nov 01 22:25:16.315536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.315505    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"b92da0b42466ae9e6e5947116fff792542e52a8c0f0f6bf29c3e5bbc97500c60"}
Nov 01 22:25:16.315976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.315576    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"ff34eb0ac4a5dd1b77c89f448170a33cc5a0e496ee71bc1c760865ba7ed6ab6d"}
Nov 01 22:25:16.315976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.315604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"5d99dda245782f8564c3ba7ae5ac4b9bb77fc3d4cf5af21de0e23f22029d7eda"}
Nov 01 22:25:16.315976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.315637    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"a190263e9c77de06e6e3389dbafa378c78941e2ba34a4dd68bbf71222d0f10a3"}
Nov 01 22:25:16.315976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.315660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"29e5d622ca58ee022f9c30a4b0163da9ed7a90218b6905eb9aa197503e3f5151"}
Nov 01 22:25:16.315976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.315693    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c3075de8-ae16-4b2d-9199-d31c4726025f","Type":"ContainerStarted","Data":"73562327ed7b3b530267c33a6233a3866195307b258ed8d99db239b34a8cdfc2"}
Nov 01 22:25:16.430260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:16.430210    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.413852274 podStartE2EDuration="3.430195971s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.343697ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:25:16.429290517 +0000 UTC m=+97.746940529" watchObservedRunningTime="2026-11-01 22:25:16.430195971 +0000 UTC m=+97.747845989"
Nov 01 22:25:17.119201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:17.119164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:17.328524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:17.328496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:18.551649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:18.551616    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:18.552070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:18.551706    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:18.556008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:18.555985    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:19.331841 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:19.331812    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b566d54b4-nwk7g"
Nov 01 22:25:19.508843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:19.508812    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c875b49-2srtf"]
Nov 01 22:25:27.657670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:27.657639    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:27.834375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:27.834337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:28.012595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:28.012568    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:36.343127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:36.343090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:25:42.523745 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:42.523695    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:25:44.527862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.527829    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-54c875b49-2srtf" podUID="35eaa797-aa6b-4128-889e-1ba55351ecda" containerName="console" containerID="cri-o://80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39" gracePeriod=15
Nov 01 22:25:44.766323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.766301    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:25:44.809712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809649    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.809712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809686    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7\" (UniqueName: \"kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.809906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809720    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp\" (UniqueName: \"kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.809906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809779    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.809906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809803    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.809906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809838    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.809906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809884    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.810173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.809923    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config\") pod \"35eaa797-aa6b-4128-889e-1ba55351ecda\" (UID: \"35eaa797-aa6b-4128-889e-1ba55351ecda\") "
Nov 01 22:25:44.810173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810059    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:25:44.810173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810114    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:44.810333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810194    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.810333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810265    4531 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35eaa797-aa6b-4128-889e-1ba55351ecda-tmp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.810431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810369    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:44.810534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810503    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:44.810646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.810626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:25:44.812182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.812153    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:44.812283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.812247    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "kube-api-access-5tvf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:25:44.812283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.812255    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config" pod "35eaa797-aa6b-4128-889e-1ba55351ecda" (UID: "35eaa797-aa6b-4128-889e-1ba55351ecda"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:25:44.911051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.911027    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-trusted-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.911051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.911051    4531 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-oauth-serving-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.911193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.911062    4531 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-serving-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.911193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.911071    4531 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35eaa797-aa6b-4128-889e-1ba55351ecda-console-oauth-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.911193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.911081    4531 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35eaa797-aa6b-4128-889e-1ba55351ecda-console-config\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:44.911193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:44.911089    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tvf7\" (UniqueName: \"kubernetes.io/projected/35eaa797-aa6b-4128-889e-1ba55351ecda-kube-api-access-5tvf7\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:25:45.424187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.424153    4531 generic.go:356] "Generic (PLEG): container finished" podID="35eaa797-aa6b-4128-889e-1ba55351ecda" containerID="80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39" exitCode=0
Nov 01 22:25:45.424297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.424221    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54c875b49-2srtf"
Nov 01 22:25:45.424297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.424238    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c875b49-2srtf" event={"ID":"35eaa797-aa6b-4128-889e-1ba55351ecda","Type":"ContainerDied","Data":"80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39"}
Nov 01 22:25:45.424380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.424318    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54c875b49-2srtf" event={"ID":"35eaa797-aa6b-4128-889e-1ba55351ecda","Type":"ContainerDied","Data":"40963261ff770b1c38c2d037778801d71a0e862a11e61adf554bdf63c4fa14b8"}
Nov 01 22:25:45.424380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.424339    4531 scope.go:122] "RemoveContainer" containerID="80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39"
Nov 01 22:25:45.432073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.432058    4531 scope.go:122] "RemoveContainer" containerID="80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39"
Nov 01 22:25:45.432331 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:25:45.432313    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39\": container with ID starting with 80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39 not found: ID does not exist" containerID="80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39"
Nov 01 22:25:45.432387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.432338    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39"} err="failed to get container status \"80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39\": rpc error: code = NotFound desc = could not find container \"80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39\": container with ID starting with 80c50f99fbe51f6e40ceb8e6ca13856d698a18def41b5fffac2c68fc163fbf39 not found: ID does not exist"
Nov 01 22:25:45.576718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.576694    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-54c875b49-2srtf"]
Nov 01 22:25:45.623277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:45.623251    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-54c875b49-2srtf"]
Nov 01 22:25:46.813667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:25:46.813633    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35eaa797-aa6b-4128-889e-1ba55351ecda" path="/var/lib/kubelet/pods/35eaa797-aa6b-4128-889e-1ba55351ecda/volumes"
Nov 01 22:26:12.524402 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:26:12.524366    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:26:20.554991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:20.554920    4531 generic.go:356] "Generic (PLEG): container finished" podID="89590d48-5c72-4c96-a46f-9d3a2a809810" containerID="ba72debe6aed0f9c1e1c7078ee83fef1e699d99fe606ffd7728239b29d01e274" exitCode=0
Nov 01 22:26:20.555346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:20.554985    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mpnn9-p62lt" event={"ID":"89590d48-5c72-4c96-a46f-9d3a2a809810","Type":"ContainerDied","Data":"ba72debe6aed0f9c1e1c7078ee83fef1e699d99fe606ffd7728239b29d01e274"}
Nov 01 22:26:21.693714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.693690    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:26:21.776279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776258    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path\") pod \"89590d48-5c72-4c96-a46f-9d3a2a809810\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") "
Nov 01 22:26:21.776410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776323    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle\") pod \"89590d48-5c72-4c96-a46f-9d3a2a809810\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") "
Nov 01 22:26:21.776410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle\") pod \"89590d48-5c72-4c96-a46f-9d3a2a809810\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") "
Nov 01 22:26:21.776410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776366    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj\" (UniqueName: \"kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj\") pod \"89590d48-5c72-4c96-a46f-9d3a2a809810\" (UID: \"89590d48-5c72-4c96-a46f-9d3a2a809810\") "
Nov 01 22:26:21.776574 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path" pod "89590d48-5c72-4c96-a46f-9d3a2a809810" (UID: "89590d48-5c72-4c96-a46f-9d3a2a809810"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:26:21.776721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle" pod "89590d48-5c72-4c96-a46f-9d3a2a809810" (UID: "89590d48-5c72-4c96-a46f-9d3a2a809810"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:21.776809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.776714    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle" pod "89590d48-5c72-4c96-a46f-9d3a2a809810" (UID: "89590d48-5c72-4c96-a46f-9d3a2a809810"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:26:21.778268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.778243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj" pod "89590d48-5c72-4c96-a46f-9d3a2a809810" (UID: "89590d48-5c72-4c96-a46f-9d3a2a809810"). InnerVolumeSpecName "kube-api-access-xgcrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:26:21.876944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.876894    4531 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-trusted-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:21.876944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.876915    4531 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/89590d48-5c72-4c96-a46f-9d3a2a809810-service-ca-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:21.876944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.876925    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgcrj\" (UniqueName: \"kubernetes.io/projected/89590d48-5c72-4c96-a46f-9d3a2a809810-kube-api-access-xgcrj\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:21.876944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:21.876935    4531 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/89590d48-5c72-4c96-a46f-9d3a2a809810-archives-path\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:26:22.567299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:22.567276    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-mpnn9-p62lt"
Nov 01 22:26:22.567474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:22.567271    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-mpnn9-p62lt" event={"ID":"89590d48-5c72-4c96-a46f-9d3a2a809810","Type":"ContainerDied","Data":"82c3617aa90afdcf6efe05567c903ede2dd28a5f16d0432098267441562ca90b"}
Nov 01 22:26:22.567474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:22.567399    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c3617aa90afdcf6efe05567c903ede2dd28a5f16d0432098267441562ca90b"
Nov 01 22:26:23.749108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:23.749060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:26:23.934084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:23.934055    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:26:28.299201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:28.299167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:26:28.526691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:28.526662    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:26:42.524844 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:26:42.524810    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:26:54.352799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:54.352765    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:26:57.326854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:26:57.326823    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:08.115286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:08.115251    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:12.525091 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:27:12.525057    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:27:28.593042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:28.592963    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:41.548643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:41.548611    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:41.898744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:41.898651    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:42.528298 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:27:42.528260    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:27:42.933358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933332    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"]
Nov 01 22:27:42.933686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933617    4531 cpu_manager.go:414] "containerMap: removing container" podUID="89590d48-5c72-4c96-a46f-9d3a2a809810" containerName="insights-gathering"
Nov 01 22:27:42.933686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933627    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89590d48-5c72-4c96-a46f-9d3a2a809810" containerName="insights-gathering"
Nov 01 22:27:42.933686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933650    4531 cpu_manager.go:414] "containerMap: removing container" podUID="35eaa797-aa6b-4128-889e-1ba55351ecda" containerName="console"
Nov 01 22:27:42.933686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933655    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35eaa797-aa6b-4128-889e-1ba55351ecda" containerName="console"
Nov 01 22:27:42.933842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933708    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="35eaa797-aa6b-4128-889e-1ba55351ecda" containerName="console"
Nov 01 22:27:42.933842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.933719    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="89590d48-5c72-4c96-a46f-9d3a2a809810" containerName="insights-gathering"
Nov 01 22:27:42.936789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:42.936771    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.016658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.016637    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.016796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.016675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6pk\" (UniqueName: \"kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.016796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.016708    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.038446 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.038425    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 01 22:27:43.038532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.038480    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 01 22:27:43.038576 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.038544    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-qxs44\""
Nov 01 22:27:43.039684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.039667    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"]
Nov 01 22:27:43.117399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.117378    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.117515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.117410    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6pk\" (UniqueName: \"kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.117515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.117436    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.117768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.117745    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.117833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.117774    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.172474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.172418    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6pk\" (UniqueName: \"kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.252262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.252241    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:27:43.372004 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.371950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:43.558129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.558097    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:43.617316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.617290    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"]
Nov 01 22:27:43.624117 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:43.624090    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee9ba54_701f_4559_be31_3e3df22f8c50.slice/crio-06ecc9b262b06fe285216365ad07a172799f4d4ec84ecffde0682969bddbd779 WatchSource:0}: Error finding container 06ecc9b262b06fe285216365ad07a172799f4d4ec84ecffde0682969bddbd779: Status 404 returned error can't find the container with id 06ecc9b262b06fe285216365ad07a172799f4d4ec84ecffde0682969bddbd779
Nov 01 22:27:43.784367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.784334    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"]
Nov 01 22:27:43.788811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.788794    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.822551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.822502    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.822551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.822546    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.822677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.822576    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4kds\" (UniqueName: \"kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.839935 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.839914    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-k7xg9\""
Nov 01 22:27:43.844811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.844794    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"]
Nov 01 22:27:43.849411 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.849386    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv" event={"ID":"aee9ba54-701f-4559-be31-3e3df22f8c50","Type":"ContainerStarted","Data":"06ecc9b262b06fe285216365ad07a172799f4d4ec84ecffde0682969bddbd779"}
Nov 01 22:27:43.922907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.922888    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.923010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.922933    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4kds\" (UniqueName: \"kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.923084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.923039    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.923347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.923321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.923440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.923371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:43.981566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:43.981542    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4kds\" (UniqueName: \"kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:44.104227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.104166    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:27:44.456938 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.456911    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"]
Nov 01 22:27:44.460637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.460619    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.468254 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.468233    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"]
Nov 01 22:27:44.474596 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:44.474572    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3aa762af_3281_4f40_bfb6_806acb624863.slice/crio-6efc6689945f1ce596f7272d951e8aa0e5d0dc204b65b5e65f7577251fa14f63 WatchSource:0}: Error finding container 6efc6689945f1ce596f7272d951e8aa0e5d0dc204b65b5e65f7577251fa14f63: Status 404 returned error can't find the container with id 6efc6689945f1ce596f7272d951e8aa0e5d0dc204b65b5e65f7577251fa14f63
Nov 01 22:27:44.509190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.509164    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-n89x5\""
Nov 01 22:27:44.517288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.517267    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"]
Nov 01 22:27:44.528301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.528282    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.528395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.528323    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4tnh\" (UniqueName: \"kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.528439 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.528405    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.564293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.564269    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"]
Nov 01 22:27:44.567656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.567638    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.616784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.616752    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-8q8ts\""
Nov 01 22:27:44.627518 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.627495    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"]
Nov 01 22:27:44.629611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.629590    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4tnh\" (UniqueName: \"kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.629717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.629632    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.629717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.629673    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.629845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.629756    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8v95\" (UniqueName: \"kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.629845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.629829    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.629950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.629905    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.630083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.630061    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.630208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.630190    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.661512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.661492    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"]
Nov 01 22:27:44.665019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.665004    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.684892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.684864    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4tnh\" (UniqueName: \"kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.717136 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.717090    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-mdf5j\""
Nov 01 22:27:44.720162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.720142    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"]
Nov 01 22:27:44.731211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731189    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.731311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731250    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.731385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731319    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.731385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731342    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5gsk\" (UniqueName: \"kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.731385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731377    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.731605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731538    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8v95\" (UniqueName: \"kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.731663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731640    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.731716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.731646    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.755686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.755664    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"]
Nov 01 22:27:44.759140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.759123    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.777471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.777453    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:27:44.785265 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.785248    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8v95\" (UniqueName: \"kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.814555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.814525    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:44.815392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.815367    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"]
Nov 01 22:27:44.817630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.817265    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-rhxpb\""
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833049    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zbx9\" (UniqueName: \"kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833158    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833324    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833476    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833571    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833768    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.837859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.833508    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5gsk\" (UniqueName: \"kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.855294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.855267    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" event={"ID":"3aa762af-3281-4f40-bfb6-806acb624863","Type":"ContainerStarted","Data":"6efc6689945f1ce596f7272d951e8aa0e5d0dc204b65b5e65f7577251fa14f63"}
Nov 01 22:27:44.888363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.888341    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:27:44.893176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.893150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5gsk\" (UniqueName: \"kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.934587 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.934560    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.934753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.934660    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zbx9\" (UniqueName: \"kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.934753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.934705    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.934962 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.934942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.935046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.935032    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:44.979748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.979682    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:27:44.990171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:44.990150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zbx9\" (UniqueName: \"kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:45.074708 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.074681    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:27:45.101920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.101891    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"]
Nov 01 22:27:45.105988 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.105967    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.136040 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.136011    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.136133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.136103    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfwz\" (UniqueName: \"kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.136177 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.136137    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.158511 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.158488    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-2fccg\""
Nov 01 22:27:45.166129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.166107    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"]
Nov 01 22:27:45.178199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.178175    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"]
Nov 01 22:27:45.185294 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:45.185264    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbeb220f_1e86_4dbf_b29e_841368dc64c3.slice/crio-70d2f6677c081820a02a8fc5cc9843021cc59fbcc2d775a2655d317201fab55a WatchSource:0}: Error finding container 70d2f6677c081820a02a8fc5cc9843021cc59fbcc2d775a2655d317201fab55a: Status 404 returned error can't find the container with id 70d2f6677c081820a02a8fc5cc9843021cc59fbcc2d775a2655d317201fab55a
Nov 01 22:27:45.236630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.236578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfwz\" (UniqueName: \"kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.236630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.236614    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.236796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.236671    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.237039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.237023    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.237524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.237503    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.257741 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.257708    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"]
Nov 01 22:27:45.264448 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:45.264427    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf471ae5d_f7e6_4fa0_8b22_e521a7c756cb.slice/crio-524a20d6fe817b3d532a1a378c033a29b73cce510de58bcf0b20b1b44eb3d61a WatchSource:0}: Error finding container 524a20d6fe817b3d532a1a378c033a29b73cce510de58bcf0b20b1b44eb3d61a: Status 404 returned error can't find the container with id 524a20d6fe817b3d532a1a378c033a29b73cce510de58bcf0b20b1b44eb3d61a
Nov 01 22:27:45.291196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.291170    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfwz\" (UniqueName: \"kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.352829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.352771    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"]
Nov 01 22:27:45.361856 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:45.361825    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f50cabc_7665_4200_9706_c857a6391b3c.slice/crio-6a16a2c9ba33550adcb42b30c4963b50175c714991913581ae32f9ad0f704fc4 WatchSource:0}: Error finding container 6a16a2c9ba33550adcb42b30c4963b50175c714991913581ae32f9ad0f704fc4: Status 404 returned error can't find the container with id 6a16a2c9ba33550adcb42b30c4963b50175c714991913581ae32f9ad0f704fc4
Nov 01 22:27:45.423596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.423566    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:27:45.448074 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.448051    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"]
Nov 01 22:27:45.454444 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:45.454414    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb3a3e5_aaee_4af7_bdf1_83050ca2b395.slice/crio-04657db229db0d9689e87ddd7674da0d46a0821a0d6247a2ecbf3e94c9e7adea WatchSource:0}: Error finding container 04657db229db0d9689e87ddd7674da0d46a0821a0d6247a2ecbf3e94c9e7adea: Status 404 returned error can't find the container with id 04657db229db0d9689e87ddd7674da0d46a0821a0d6247a2ecbf3e94c9e7adea
Nov 01 22:27:45.800822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.800777    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"]
Nov 01 22:27:45.807275 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:45.807235    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f682184_7ca2_47b5_ae61_d14198902cde.slice/crio-a9e8af6f79d434c26ff0243a403c996304237f9cdcd9df4afa122388430324dc WatchSource:0}: Error finding container a9e8af6f79d434c26ff0243a403c996304237f9cdcd9df4afa122388430324dc: Status 404 returned error can't find the container with id a9e8af6f79d434c26ff0243a403c996304237f9cdcd9df4afa122388430324dc
Nov 01 22:27:45.861609 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.861548    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97" event={"ID":"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb","Type":"ContainerStarted","Data":"524a20d6fe817b3d532a1a378c033a29b73cce510de58bcf0b20b1b44eb3d61a"}
Nov 01 22:27:45.863017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.862988    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr" event={"ID":"bbeb220f-1e86-4dbf-b29e-841368dc64c3","Type":"ContainerStarted","Data":"70d2f6677c081820a02a8fc5cc9843021cc59fbcc2d775a2655d317201fab55a"}
Nov 01 22:27:45.865175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.865145    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" event={"ID":"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395","Type":"ContainerStarted","Data":"04657db229db0d9689e87ddd7674da0d46a0821a0d6247a2ecbf3e94c9e7adea"}
Nov 01 22:27:45.866530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.866505    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" event={"ID":"2f50cabc-7665-4200-9706-c857a6391b3c","Type":"ContainerStarted","Data":"6a16a2c9ba33550adcb42b30c4963b50175c714991913581ae32f9ad0f704fc4"}
Nov 01 22:27:45.868006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:45.867978    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw" event={"ID":"5f682184-7ca2-47b5-ae61-d14198902cde","Type":"ContainerStarted","Data":"a9e8af6f79d434c26ff0243a403c996304237f9cdcd9df4afa122388430324dc"}
Nov 01 22:27:48.205564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.205489    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"]
Nov 01 22:27:48.210118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.210094    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.264197 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.264127    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"]
Nov 01 22:27:48.270050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.269778    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.270050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.269923    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.270050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.269953    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhp5\" (UniqueName: \"kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.372585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.371016    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.372585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.371188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhp5\" (UniqueName: \"kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.372585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.371235    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.372585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.371805    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.372585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.372210    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.446646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.446573    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhp5\" (UniqueName: \"kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.537579 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.537510    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:27:48.957125 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:48.957094    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"]
Nov 01 22:27:49.895413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:49.895345    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:49.905383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:49.905306    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46" event={"ID":"d9a4ff41-bd6b-4386-8bdf-fb356e84909c","Type":"ContainerStarted","Data":"78d2059e166b2fcb3551b592bd55c34fc8775cc4833dfbe700ac3dcde54cd9c8"}
Nov 01 22:27:50.201413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.201380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:50.862319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.862278    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"]
Nov 01 22:27:50.866716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.866694    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:50.901436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.901413    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"]
Nov 01 22:27:50.906932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.906914    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:50.948814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.948785    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"]
Nov 01 22:27:50.950701 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.950657    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"]
Nov 01 22:27:50.954665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.954647    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:50.966318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.966141    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"]
Nov 01 22:27:50.970183 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.970159    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"]
Nov 01 22:27:50.982026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.982004    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:50.995685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995660    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:50.995814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995714    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzk9n\" (UniqueName: \"kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:50.995814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995762    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:50.995934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995845    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:50.995934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:50.996041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995930    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:50.996041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995959    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtsgn\" (UniqueName: \"kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:50.996041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.995998    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:50.996192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.996040    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz89d\" (UniqueName: \"kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:50.999558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:50.999536    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"]
Nov 01 22:27:51.003526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.003502    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.015800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.015777    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"]
Nov 01 22:27:51.033372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.033305    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"]
Nov 01 22:27:51.052766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.052683    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bv4kj\""
Nov 01 22:27:51.053024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.052909    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 22:27:51.053024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.052997    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 22:27:51.053177 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.053054    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 22:27:51.069509 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.069485    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"]
Nov 01 22:27:51.096532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096489    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz89d\" (UniqueName: \"kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.096667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096563    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.096774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096709    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.096774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096755    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.096908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096789    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzk9n\" (UniqueName: \"kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.096908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096817    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.096908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096871    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.096908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096903    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm6ct\" (UniqueName: \"kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.097120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096946    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.097120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.096979    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.097120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097022    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.097120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097048    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5dr\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.097120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097076    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.097120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097109    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097128    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097153    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097199    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097241    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtsgn\" (UniqueName: \"kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097275    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097310    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097339    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097367    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.097431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097416    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.097981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097692    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.097981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.097938    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.099167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.098263    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.099167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.098454    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.142946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.142800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:51.160922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.160888    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtsgn\" (UniqueName: \"kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.162427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.162393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz89d\" (UniqueName: \"kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.167075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.166800    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzk9n\" (UniqueName: \"kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.188582 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.188559    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:27:51.199194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.198636    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.199194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.198707    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.199194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.199080    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.199873    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.199940    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.199974    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm6ct\" (UniqueName: \"kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200015    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200094    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5dr\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200121    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200150    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200201    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200232    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.200814    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.201049    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.201496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.201718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.202158    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.203134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.202771    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.213557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.211744    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.237580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.237528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.238659 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.238470    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:27:51.258542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.258521    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm6ct\" (UniqueName: \"kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.258990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.258956    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.261513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.261468    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5dr\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr\") pod \"maas-default-gateway-openshift-default-7f694df8bc-dss7f\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.277470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.277328    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:27:51.306917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.306183    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"]
Nov 01 22:27:51.312764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.312129    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:27:51.312764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.312272    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.329877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.326058    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:27:51.361362 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.361307    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-ffq9r\""
Nov 01 22:27:51.367262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.367193    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"]
Nov 01 22:27:51.396119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.396022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402182    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402278    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402315    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402352    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402424    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402479    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402541    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.402866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.402586    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9b2t\" (UniqueName: \"kubernetes.io/projected/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-kube-api-access-h9b2t\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504136    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504194    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504229    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504261    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504345    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504391    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504445    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.504498    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9b2t\" (UniqueName: \"kubernetes.io/projected/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-kube-api-access-h9b2t\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.505536    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.505815    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.506301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.506905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.506517    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.508073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.507916    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.510840 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.510798    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.512366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.512347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.561557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.561517    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9b2t\" (UniqueName: \"kubernetes.io/projected/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-kube-api-access-h9b2t\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.564820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.564778    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6\" (UID: \"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.582479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.582455    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:27:51.604096 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.604063    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"]
Nov 01 22:27:51.616266 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:51.616234    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3ebe7b_dbe7_4f56_b95d_cad8d3ae8b35.slice/crio-25938861985d79a0140f367a75000b806dc753165bc4b70af076aef6227e7cf1 WatchSource:0}: Error finding container 25938861985d79a0140f367a75000b806dc753165bc4b70af076aef6227e7cf1: Status 404 returned error can't find the container with id 25938861985d79a0140f367a75000b806dc753165bc4b70af076aef6227e7cf1
Nov 01 22:27:51.654225 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.654110    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:27:51.707317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.707242    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"]
Nov 01 22:27:51.715429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.714981    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"]
Nov 01 22:27:51.729434 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:51.729407    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1c0771_c9eb_4322_aa79_a85ec4a3a4fc.slice/crio-b335ca3ae6d3855d9ad1c6c24aa2ba91c44ac6e21aefccb095f0a1f646c4fb7a WatchSource:0}: Error finding container b335ca3ae6d3855d9ad1c6c24aa2ba91c44ac6e21aefccb095f0a1f646c4fb7a: Status 404 returned error can't find the container with id b335ca3ae6d3855d9ad1c6c24aa2ba91c44ac6e21aefccb095f0a1f646c4fb7a
Nov 01 22:27:51.773758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.771588    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"]
Nov 01 22:27:51.829671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.829647    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"]
Nov 01 22:27:51.949680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.949591    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r" event={"ID":"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc","Type":"ContainerStarted","Data":"b335ca3ae6d3855d9ad1c6c24aa2ba91c44ac6e21aefccb095f0a1f646c4fb7a"}
Nov 01 22:27:51.959899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.959805    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" event={"ID":"747f48c3-52de-4ed9-a5b7-ea0d3960fd40","Type":"ContainerStarted","Data":"8dda84c77d2f0d8770140b8f2b4695768944b434a97e5cacef702ba9af6fc51e"}
Nov 01 22:27:51.972414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.972376    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx" event={"ID":"03722b82-e68a-4dda-9cea-aece7a654f72","Type":"ContainerStarted","Data":"2c853062a1726959d3bdfe543037169a101efb2788995e70f724ade52d538f16"}
Nov 01 22:27:51.981383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.981354    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff" event={"ID":"70c2e8cb-40ff-4488-bb99-87b7fdb9922c","Type":"ContainerStarted","Data":"ce9588d11c2731a87a194d3233c254d17899f6162163baeaba256a24fa73e611"}
Nov 01 22:27:51.985331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:51.985265    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4" event={"ID":"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35","Type":"ContainerStarted","Data":"25938861985d79a0140f367a75000b806dc753165bc4b70af076aef6227e7cf1"}
Nov 01 22:27:52.076212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:52.076147    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"]
Nov 01 22:27:52.095081 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:27:52.095051    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45d6b2e_944f_49a7_be3d_4b6dd7a0dd85.slice/crio-dc9b792e40188576e3b874f93e7dad6e9a3bf446675ff1063a6905eefdb0f773 WatchSource:0}: Error finding container dc9b792e40188576e3b874f93e7dad6e9a3bf446675ff1063a6905eefdb0f773: Status 404 returned error can't find the container with id dc9b792e40188576e3b874f93e7dad6e9a3bf446675ff1063a6905eefdb0f773
Nov 01 22:27:53.008959 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:53.008921    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6" event={"ID":"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85","Type":"ContainerStarted","Data":"dc9b792e40188576e3b874f93e7dad6e9a3bf446675ff1063a6905eefdb0f773"}
Nov 01 22:27:55.107567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:27:55.107499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:05.791464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:05.790899    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:05.791464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:05.790975    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:05.791464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:05.791011    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:06.234297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:06.232795    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" event={"ID":"747f48c3-52de-4ed9-a5b7-ea0d3960fd40","Type":"ContainerStarted","Data":"66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"}
Nov 01 22:28:06.329051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:06.328212    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:28:06.329358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:06.329305    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:28:06.417217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:06.416115    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podStartSLOduration=2.452553233 podStartE2EDuration="16.416095434s" totalImagesPullingTime="13.963542201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:06.390097106 +0000 UTC m=+267.707747107" watchObservedRunningTime="2026-11-01 22:28:06.416095434 +0000 UTC m=+267.733745427"
Nov 01 22:28:07.245593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.245564    4531 generic.go:356] "Generic (PLEG): container finished" podID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerID="cf6c9a1d23aa2039600336d9d67ee7ff91bf10dea985c7ad7833741aca92a43f" exitCode=0
Nov 01 22:28:07.246006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.245645    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv" event={"ID":"aee9ba54-701f-4559-be31-3e3df22f8c50","Type":"ContainerDied","Data":"cf6c9a1d23aa2039600336d9d67ee7ff91bf10dea985c7ad7833741aca92a43f"}
Nov 01 22:28:07.251532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.250271    4531 generic.go:356] "Generic (PLEG): container finished" podID="3aa762af-3281-4f40-bfb6-806acb624863" containerID="42fdf533e6724d2daeddd186454cfcc48f846d19e10146455053535cfc731571" exitCode=0
Nov 01 22:28:07.251532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.251480    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" event={"ID":"3aa762af-3281-4f40-bfb6-806acb624863","Type":"ContainerDied","Data":"42fdf533e6724d2daeddd186454cfcc48f846d19e10146455053535cfc731571"}
Nov 01 22:28:07.334713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.334683    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:28:07.914526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.914490    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:07.914638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.914561    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:07.914638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:07.914593    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:28:08.258521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.258477    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6" event={"ID":"d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85","Type":"ContainerStarted","Data":"0782142e7d6aeeb094da72755f4fdd063d6cc485a3b1b0271ab15bf0185c5f98"}
Nov 01 22:28:08.260308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.260133    4531 generic.go:356] "Generic (PLEG): container finished" podID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerID="3fcb359ded39eac1a18ecf61d49d2bf5f14e4ff05012e92ae7c90b55670cc551" exitCode=0
Nov 01 22:28:08.260308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.260260    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4" event={"ID":"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35","Type":"ContainerDied","Data":"3fcb359ded39eac1a18ecf61d49d2bf5f14e4ff05012e92ae7c90b55670cc551"}
Nov 01 22:28:08.262537 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.262511    4531 generic.go:356] "Generic (PLEG): container finished" podID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerID="d03af9795aa8297c1a5b1e74c6cce1a748cb62ac923444805abb76f8a088dbc1" exitCode=0
Nov 01 22:28:08.262635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.262593    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r" event={"ID":"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc","Type":"ContainerDied","Data":"d03af9795aa8297c1a5b1e74c6cce1a748cb62ac923444805abb76f8a088dbc1"}
Nov 01 22:28:08.263816 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.263795    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:28:08.367860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.367817    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6" podStartSLOduration=1.56105956 podStartE2EDuration="17.367803441s" totalImagesPullingTime="15.806743881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:08.367794796 +0000 UTC m=+269.685444800" watchObservedRunningTime="2026-11-01 22:28:08.367803441 +0000 UTC m=+269.685453439"
Nov 01 22:28:08.655069 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.654997    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:28:08.660411 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:08.660386    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:28:09.268229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.268197    4531 generic.go:356] "Generic (PLEG): container finished" podID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerID="a024253c8d00e093e490b4d5fbbfe3e2c864b0c5b460f0ec6d578cebbc1f678a" exitCode=0
Nov 01 22:28:09.268620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.268268    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97" event={"ID":"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb","Type":"ContainerDied","Data":"a024253c8d00e093e490b4d5fbbfe3e2c864b0c5b460f0ec6d578cebbc1f678a"}
Nov 01 22:28:09.270297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.270271    4531 generic.go:356] "Generic (PLEG): container finished" podID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerID="5fd3b264a5985e0083861f07f529b564a1f9e683a07d2fbfe54c9a76eea8fa14" exitCode=0
Nov 01 22:28:09.270389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.270320    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr" event={"ID":"bbeb220f-1e86-4dbf-b29e-841368dc64c3","Type":"ContainerDied","Data":"5fd3b264a5985e0083861f07f529b564a1f9e683a07d2fbfe54c9a76eea8fa14"}
Nov 01 22:28:09.271985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.271960    4531 generic.go:356] "Generic (PLEG): container finished" podID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerID="9564fb23081194b52425da9f2ea2b7966660e0251952de60cb84ebb0852e4a05" exitCode=0
Nov 01 22:28:09.272073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.271987    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" event={"ID":"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395","Type":"ContainerDied","Data":"9564fb23081194b52425da9f2ea2b7966660e0251952de60cb84ebb0852e4a05"}
Nov 01 22:28:09.273406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.273389    4531 generic.go:356] "Generic (PLEG): container finished" podID="2f50cabc-7665-4200-9706-c857a6391b3c" containerID="4734bad7733b3a73f055fc986baffa7e3f38656cffbe89fd1d6506c016a0faaa" exitCode=0
Nov 01 22:28:09.273490 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.273456    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" event={"ID":"2f50cabc-7665-4200-9706-c857a6391b3c","Type":"ContainerDied","Data":"4734bad7733b3a73f055fc986baffa7e3f38656cffbe89fd1d6506c016a0faaa"}
Nov 01 22:28:09.274966 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.274943    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw" event={"ID":"5f682184-7ca2-47b5-ae61-d14198902cde","Type":"ContainerStarted","Data":"a51ee687c6180fcb40662731c97b8b6534fca402f259e6fa50a7d2dbafc859c6"}
Nov 01 22:28:09.275458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.275436    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:28:09.276363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:09.276345    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6"
Nov 01 22:28:10.281875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.281848    4531 generic.go:356] "Generic (PLEG): container finished" podID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerID="461b7363d243fb99919b0df18153e8efee585c12661aee69f07bfd0d215c1759" exitCode=0
Nov 01 22:28:10.282245 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.281929    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv" event={"ID":"aee9ba54-701f-4559-be31-3e3df22f8c50","Type":"ContainerDied","Data":"461b7363d243fb99919b0df18153e8efee585c12661aee69f07bfd0d215c1759"}
Nov 01 22:28:10.283534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.283508    4531 generic.go:356] "Generic (PLEG): container finished" podID="5f682184-7ca2-47b5-ae61-d14198902cde" containerID="a51ee687c6180fcb40662731c97b8b6534fca402f259e6fa50a7d2dbafc859c6" exitCode=0
Nov 01 22:28:10.283641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.283594    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw" event={"ID":"5f682184-7ca2-47b5-ae61-d14198902cde","Type":"ContainerDied","Data":"a51ee687c6180fcb40662731c97b8b6534fca402f259e6fa50a7d2dbafc859c6"}
Nov 01 22:28:10.285428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.285372    4531 generic.go:356] "Generic (PLEG): container finished" podID="03722b82-e68a-4dda-9cea-aece7a654f72" containerID="5556bd8d62e06f15f53e60f2524ce8da5bbf752cb93ca308b35dd31b73105916" exitCode=0
Nov 01 22:28:10.285428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.285414    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx" event={"ID":"03722b82-e68a-4dda-9cea-aece7a654f72","Type":"ContainerDied","Data":"5556bd8d62e06f15f53e60f2524ce8da5bbf752cb93ca308b35dd31b73105916"}
Nov 01 22:28:10.287133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.287112    4531 generic.go:356] "Generic (PLEG): container finished" podID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerID="4109cf8634f74c25e15a9a4ea1d510b97bf84f5f7d4ed89ce122392993ce09d2" exitCode=0
Nov 01 22:28:10.287220 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.287180    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46" event={"ID":"d9a4ff41-bd6b-4386-8bdf-fb356e84909c","Type":"ContainerDied","Data":"4109cf8634f74c25e15a9a4ea1d510b97bf84f5f7d4ed89ce122392993ce09d2"}
Nov 01 22:28:10.288776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.288757    4531 generic.go:356] "Generic (PLEG): container finished" podID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerID="24821ced9b52fd54e6d70f8948872a56219b2c64a20f9c2559a003e904e16a15" exitCode=0
Nov 01 22:28:10.288845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:10.288823    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff" event={"ID":"70c2e8cb-40ff-4488-bb99-87b7fdb9922c","Type":"ContainerDied","Data":"24821ced9b52fd54e6d70f8948872a56219b2c64a20f9c2559a003e904e16a15"}
Nov 01 22:28:11.295852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.295793    4531 generic.go:356] "Generic (PLEG): container finished" podID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerID="28ec289d57380de8e4ae2cb46b68b646269f06b13fb7877f4a9d0bfa0b9ecdb7" exitCode=0
Nov 01 22:28:11.296344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.295867    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" event={"ID":"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395","Type":"ContainerDied","Data":"28ec289d57380de8e4ae2cb46b68b646269f06b13fb7877f4a9d0bfa0b9ecdb7"}
Nov 01 22:28:11.297932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.297910    4531 generic.go:356] "Generic (PLEG): container finished" podID="2f50cabc-7665-4200-9706-c857a6391b3c" containerID="e17e57a1adddc503f8f0f56057cad8b050d36cc66d00ef0e850e4cee6e2e7794" exitCode=0
Nov 01 22:28:11.298023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.298001    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" event={"ID":"2f50cabc-7665-4200-9706-c857a6391b3c","Type":"ContainerDied","Data":"e17e57a1adddc503f8f0f56057cad8b050d36cc66d00ef0e850e4cee6e2e7794"}
Nov 01 22:28:11.299948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.299927    4531 generic.go:356] "Generic (PLEG): container finished" podID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerID="1da49e574b3692dea1c71586b6abfd3bbf526952c978c3ee100234f3c0d181c7" exitCode=0
Nov 01 22:28:11.300035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.299993    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46" event={"ID":"d9a4ff41-bd6b-4386-8bdf-fb356e84909c","Type":"ContainerDied","Data":"1da49e574b3692dea1c71586b6abfd3bbf526952c978c3ee100234f3c0d181c7"}
Nov 01 22:28:11.302114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.302016    4531 generic.go:356] "Generic (PLEG): container finished" podID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerID="b2aa16e347ac0b40d075c533d9670fc7ea3e059228484f69c63238ffa44c8c6c" exitCode=0
Nov 01 22:28:11.302114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.302098    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4" event={"ID":"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35","Type":"ContainerDied","Data":"b2aa16e347ac0b40d075c533d9670fc7ea3e059228484f69c63238ffa44c8c6c"}
Nov 01 22:28:11.303870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.303848    4531 generic.go:356] "Generic (PLEG): container finished" podID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerID="e9ad978648fa4954b8b93c0240d1987a3cf58fcbd1a11b333f7202eceff8c662" exitCode=0
Nov 01 22:28:11.303944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:11.303921    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r" event={"ID":"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc","Type":"ContainerDied","Data":"e9ad978648fa4954b8b93c0240d1987a3cf58fcbd1a11b333f7202eceff8c662"}
Nov 01 22:28:12.318329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:12.318297    4531 generic.go:356] "Generic (PLEG): container finished" podID="03722b82-e68a-4dda-9cea-aece7a654f72" containerID="2f6260b07cdffa989ada3830dd19f9cb48147889ba722b929280c56155c97d54" exitCode=0
Nov 01 22:28:12.318690 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:12.318428    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx" event={"ID":"03722b82-e68a-4dda-9cea-aece7a654f72","Type":"ContainerDied","Data":"2f6260b07cdffa989ada3830dd19f9cb48147889ba722b929280c56155c97d54"}
Nov 01 22:28:12.356635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:12.356582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:12.531512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:12.531434    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:12.541929 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:28:12.541660    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:28:13.335668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:13.334664    4531 generic.go:356] "Generic (PLEG): container finished" podID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerID="f1b401ca6cbb56e434f3a52f22eeaa29ab62464cdc3d7175c626cd6b3bf3aa29" exitCode=0
Nov 01 22:28:13.336386 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:13.336313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff" event={"ID":"70c2e8cb-40ff-4488-bb99-87b7fdb9922c","Type":"ContainerDied","Data":"f1b401ca6cbb56e434f3a52f22eeaa29ab62464cdc3d7175c626cd6b3bf3aa29"}
Nov 01 22:28:13.359815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:13.359672    4531 generic.go:356] "Generic (PLEG): container finished" podID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerID="31798b4da2ad54624c544e9af52eb1971b9c22d1341fbc068fb9fb63595e3b64" exitCode=0
Nov 01 22:28:13.359815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:13.359762    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97" event={"ID":"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb","Type":"ContainerDied","Data":"31798b4da2ad54624c544e9af52eb1971b9c22d1341fbc068fb9fb63595e3b64"}
Nov 01 22:28:13.367356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:13.366604    4531 generic.go:356] "Generic (PLEG): container finished" podID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerID="bafb57e14bb1e238d5d6e30613f3ad1ce4ff690e3f85923a4e11756e017d3251" exitCode=0
Nov 01 22:28:13.367356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:13.366671    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr" event={"ID":"bbeb220f-1e86-4dbf-b29e-841368dc64c3","Type":"ContainerDied","Data":"bafb57e14bb1e238d5d6e30613f3ad1ce4ff690e3f85923a4e11756e017d3251"}
Nov 01 22:28:16.435495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:16.435400    4531 generic.go:356] "Generic (PLEG): container finished" podID="3aa762af-3281-4f40-bfb6-806acb624863" containerID="321681b4ddcfd51df13f256069fc0e0d6fe99e47d41f3c0ed133f0b8352914e2" exitCode=0
Nov 01 22:28:16.436034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:16.435592    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" event={"ID":"3aa762af-3281-4f40-bfb6-806acb624863","Type":"ContainerDied","Data":"321681b4ddcfd51df13f256069fc0e0d6fe99e47d41f3c0ed133f0b8352914e2"}
Nov 01 22:28:19.499977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:19.499936    4531 generic.go:356] "Generic (PLEG): container finished" podID="5f682184-7ca2-47b5-ae61-d14198902cde" containerID="ff685433669823eb3e8270be8fc6285307e5ae7341588c7f6e3fecd376e2611a" exitCode=0
Nov 01 22:28:19.500458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:19.500075    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw" event={"ID":"5f682184-7ca2-47b5-ae61-d14198902cde","Type":"ContainerDied","Data":"ff685433669823eb3e8270be8fc6285307e5ae7341588c7f6e3fecd376e2611a"}
Nov 01 22:28:28.924804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:28.924666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:34.718757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:34.718590    4531 generic.go:356] "Generic (PLEG): container finished" podID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerID="e9db213ff69d9c78956df04a1203678ae12cf8cf8ad39936193716b426db4a34" exitCode=0
Nov 01 22:28:34.719720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:34.719105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv" event={"ID":"aee9ba54-701f-4559-be31-3e3df22f8c50","Type":"ContainerDied","Data":"e9db213ff69d9c78956df04a1203678ae12cf8cf8ad39936193716b426db4a34"}
Nov 01 22:28:35.737247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.737170    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" event={"ID":"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395","Type":"ContainerStarted","Data":"e6c5ac041f7d4618a391de61316692e6c5be0b0056f4c287e1a980a3f5e922fb"}
Nov 01 22:28:35.743289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.743257    4531 generic.go:356] "Generic (PLEG): container finished" podID="5f682184-7ca2-47b5-ae61-d14198902cde" containerID="012778036c7f99e918f5c9db84e0532bd0acd9f1fb7576e67dc44478ed153a40" exitCode=0
Nov 01 22:28:35.743430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.743345    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw" event={"ID":"5f682184-7ca2-47b5-ae61-d14198902cde","Type":"ContainerDied","Data":"012778036c7f99e918f5c9db84e0532bd0acd9f1fb7576e67dc44478ed153a40"}
Nov 01 22:28:35.748499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.748000    4531 generic.go:356] "Generic (PLEG): container finished" podID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerID="16a116a01d62b00547dcfce1aa50fa403d0434dd97c6a3bc656a19b981479770" exitCode=0
Nov 01 22:28:35.748499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.748454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46" event={"ID":"d9a4ff41-bd6b-4386-8bdf-fb356e84909c","Type":"ContainerDied","Data":"16a116a01d62b00547dcfce1aa50fa403d0434dd97c6a3bc656a19b981479770"}
Nov 01 22:28:35.849960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.849911    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" podStartSLOduration=3.336247744 podStartE2EDuration="51.84987827s" totalImagesPullingTime="48.513630526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:35.846915802 +0000 UTC m=+297.164565793" watchObservedRunningTime="2026-11-01 22:28:35.84987827 +0000 UTC m=+297.167528273"
Nov 01 22:28:35.963618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:35.963516    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:28:36.134927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.134182    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util\") pod \"aee9ba54-701f-4559-be31-3e3df22f8c50\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") "
Nov 01 22:28:36.134927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.134232    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk\" (UniqueName: \"kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk\") pod \"aee9ba54-701f-4559-be31-3e3df22f8c50\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") "
Nov 01 22:28:36.134927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.134326    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle\") pod \"aee9ba54-701f-4559-be31-3e3df22f8c50\" (UID: \"aee9ba54-701f-4559-be31-3e3df22f8c50\") "
Nov 01 22:28:36.140221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.140154    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle" pod "aee9ba54-701f-4559-be31-3e3df22f8c50" (UID: "aee9ba54-701f-4559-be31-3e3df22f8c50"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:36.141160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.141118    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk" pod "aee9ba54-701f-4559-be31-3e3df22f8c50" (UID: "aee9ba54-701f-4559-be31-3e3df22f8c50"). InnerVolumeSpecName "kube-api-access-xb6pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:36.143691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.143672    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util" pod "aee9ba54-701f-4559-be31-3e3df22f8c50" (UID: "aee9ba54-701f-4559-be31-3e3df22f8c50"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:36.235786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.235751    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:36.235786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.235789    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb6pk\" (UniqueName: \"kubernetes.io/projected/aee9ba54-701f-4559-be31-3e3df22f8c50-kube-api-access-xb6pk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:36.236040 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.235807    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aee9ba54-701f-4559-be31-3e3df22f8c50-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:36.765699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.765156    4531 generic.go:356] "Generic (PLEG): container finished" podID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerID="747b690b33bd90a2a42c98f9f229da8d9c1da96f5cc450d3f85544966dc22ed3" exitCode=0
Nov 01 22:28:36.765699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.765299    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r" event={"ID":"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc","Type":"ContainerDied","Data":"747b690b33bd90a2a42c98f9f229da8d9c1da96f5cc450d3f85544966dc22ed3"}
Nov 01 22:28:36.769852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.769799    4531 generic.go:356] "Generic (PLEG): container finished" podID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerID="e6c5ac041f7d4618a391de61316692e6c5be0b0056f4c287e1a980a3f5e922fb" exitCode=0
Nov 01 22:28:36.769947 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.769921    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" event={"ID":"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395","Type":"ContainerDied","Data":"e6c5ac041f7d4618a391de61316692e6c5be0b0056f4c287e1a980a3f5e922fb"}
Nov 01 22:28:36.776226 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.776143    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" event={"ID":"2f50cabc-7665-4200-9706-c857a6391b3c","Type":"ContainerStarted","Data":"912c88bed91a2aa6e88933fa9720790ee4ad3811b22f70103b136849d9a839ce"}
Nov 01 22:28:36.779927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.779908    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv"
Nov 01 22:28:36.782851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.782781    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71t9ghv" event={"ID":"aee9ba54-701f-4559-be31-3e3df22f8c50","Type":"ContainerDied","Data":"06ecc9b262b06fe285216365ad07a172799f4d4ec84ecffde0682969bddbd779"}
Nov 01 22:28:36.782851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.782829    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06ecc9b262b06fe285216365ad07a172799f4d4ec84ecffde0682969bddbd779"
Nov 01 22:28:36.972440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:36.971294    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:28:37.044221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.044197    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz\" (UniqueName: \"kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz\") pod \"5f682184-7ca2-47b5-ae61-d14198902cde\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") "
Nov 01 22:28:37.044318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.044285    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util\") pod \"5f682184-7ca2-47b5-ae61-d14198902cde\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") "
Nov 01 22:28:37.044427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.044404    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle\") pod \"5f682184-7ca2-47b5-ae61-d14198902cde\" (UID: \"5f682184-7ca2-47b5-ae61-d14198902cde\") "
Nov 01 22:28:37.045292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.045260    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle" pod "5f682184-7ca2-47b5-ae61-d14198902cde" (UID: "5f682184-7ca2-47b5-ae61-d14198902cde"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.049658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.049626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util" pod "5f682184-7ca2-47b5-ae61-d14198902cde" (UID: "5f682184-7ca2-47b5-ae61-d14198902cde"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.050153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.050128    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz" pod "5f682184-7ca2-47b5-ae61-d14198902cde" (UID: "5f682184-7ca2-47b5-ae61-d14198902cde"). InnerVolumeSpecName "kube-api-access-nzfwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:37.064249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.064211    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:28:37.146914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.146845    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.146914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.146874    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f682184-7ca2-47b5-ae61-d14198902cde-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.146914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.146890    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzfwz\" (UniqueName: \"kubernetes.io/projected/5f682184-7ca2-47b5-ae61-d14198902cde-kube-api-access-nzfwz\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.248414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.247560    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5\" (UniqueName: \"kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5\") pod \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") "
Nov 01 22:28:37.248414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.247649    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle\") pod \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") "
Nov 01 22:28:37.248414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.247681    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util\") pod \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\" (UID: \"d9a4ff41-bd6b-4386-8bdf-fb356e84909c\") "
Nov 01 22:28:37.248906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.248875    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle" pod "d9a4ff41-bd6b-4386-8bdf-fb356e84909c" (UID: "d9a4ff41-bd6b-4386-8bdf-fb356e84909c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.252001 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.251971    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5" pod "d9a4ff41-bd6b-4386-8bdf-fb356e84909c" (UID: "d9a4ff41-bd6b-4386-8bdf-fb356e84909c"). InnerVolumeSpecName "kube-api-access-dvhp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:37.253630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.253603    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util" pod "d9a4ff41-bd6b-4386-8bdf-fb356e84909c" (UID: "d9a4ff41-bd6b-4386-8bdf-fb356e84909c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:37.256418 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.255927    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" podStartSLOduration=3.560167886 podStartE2EDuration="53.255909533s" totalImagesPullingTime="49.695741647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:37.146240117 +0000 UTC m=+298.463890121" watchObservedRunningTime="2026-11-01 22:28:37.255909533 +0000 UTC m=+298.573559539"
Nov 01 22:28:37.348746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.348623    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.348746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.348656    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.348746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.348672    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvhp5\" (UniqueName: \"kubernetes.io/projected/d9a4ff41-bd6b-4386-8bdf-fb356e84909c-kube-api-access-dvhp5\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:37.788040 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.788000    4531 generic.go:356] "Generic (PLEG): container finished" podID="2f50cabc-7665-4200-9706-c857a6391b3c" containerID="912c88bed91a2aa6e88933fa9720790ee4ad3811b22f70103b136849d9a839ce" exitCode=0
Nov 01 22:28:37.788500 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.788090    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" event={"ID":"2f50cabc-7665-4200-9706-c857a6391b3c","Type":"ContainerDied","Data":"912c88bed91a2aa6e88933fa9720790ee4ad3811b22f70103b136849d9a839ce"}
Nov 01 22:28:37.790855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.790830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw" event={"ID":"5f682184-7ca2-47b5-ae61-d14198902cde","Type":"ContainerDied","Data":"a9e8af6f79d434c26ff0243a403c996304237f9cdcd9df4afa122388430324dc"}
Nov 01 22:28:37.790975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.790870    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9e8af6f79d434c26ff0243a403c996304237f9cdcd9df4afa122388430324dc"
Nov 01 22:28:37.790975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.790839    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cfq4mw"
Nov 01 22:28:37.793642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.793514    4531 generic.go:356] "Generic (PLEG): container finished" podID="03722b82-e68a-4dda-9cea-aece7a654f72" containerID="d7a97ce99aec3007798124bb4082c8a2d3e036571966725b74f76290e122cbb2" exitCode=0
Nov 01 22:28:37.793642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.793598    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx" event={"ID":"03722b82-e68a-4dda-9cea-aece7a654f72","Type":"ContainerDied","Data":"d7a97ce99aec3007798124bb4082c8a2d3e036571966725b74f76290e122cbb2"}
Nov 01 22:28:37.799329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.799307    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46" event={"ID":"d9a4ff41-bd6b-4386-8bdf-fb356e84909c","Type":"ContainerDied","Data":"78d2059e166b2fcb3551b592bd55c34fc8775cc4833dfbe700ac3dcde54cd9c8"}
Nov 01 22:28:37.799426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.799343    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78d2059e166b2fcb3551b592bd55c34fc8775cc4833dfbe700ac3dcde54cd9c8"
Nov 01 22:28:37.799426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.799410    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71g4m46"
Nov 01 22:28:37.804239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.804215    4531 generic.go:356] "Generic (PLEG): container finished" podID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerID="e80b9c3da34c0123985c620c289f11eddc445be320c7ea2ff305d5d4a9dbaa0f" exitCode=0
Nov 01 22:28:37.804614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.804584    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4" event={"ID":"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35","Type":"ContainerDied","Data":"e80b9c3da34c0123985c620c289f11eddc445be320c7ea2ff305d5d4a9dbaa0f"}
Nov 01 22:28:37.952817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.952206    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:28:37.990038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:37.990009    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:28:38.055625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.055565    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d\" (UniqueName: \"kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d\") pod \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") "
Nov 01 22:28:38.055625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.055623    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9\" (UniqueName: \"kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9\") pod \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") "
Nov 01 22:28:38.055825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.055752    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle\") pod \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") "
Nov 01 22:28:38.055825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.055794    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle\") pod \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") "
Nov 01 22:28:38.055933 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.055832    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util\") pod \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\" (UID: \"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc\") "
Nov 01 22:28:38.055997 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.055949    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util\") pod \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\" (UID: \"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395\") "
Nov 01 22:28:38.056652 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.056627    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle" pod "1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" (UID: "1eb3a3e5-aaee-4af7-bdf1-83050ca2b395"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:38.061754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.059190    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9" pod "1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" (UID: "1eb3a3e5-aaee-4af7-bdf1-83050ca2b395"). InnerVolumeSpecName "kube-api-access-7zbx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:38.061754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.061674    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle" pod "dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" (UID: "dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:38.061754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.061746    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util" pod "dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" (UID: "dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:38.061988 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.061951    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util" pod "1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" (UID: "1eb3a3e5-aaee-4af7-bdf1-83050ca2b395"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:38.062286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.062265    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d" pod "dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" (UID: "dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc"). InnerVolumeSpecName "kube-api-access-cz89d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:38.157183 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.157150    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zbx9\" (UniqueName: \"kubernetes.io/projected/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-kube-api-access-7zbx9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:38.157289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.157185    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:38.157289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.157203    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:38.157289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.157215    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:38.157289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.157229    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1eb3a3e5-aaee-4af7-bdf1-83050ca2b395-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:38.157289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.157242    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz89d\" (UniqueName: \"kubernetes.io/projected/dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc-kube-api-access-cz89d\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:38.823819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.823715    4531 generic.go:356] "Generic (PLEG): container finished" podID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerID="54076969c4a228b6f70f7f93d67e0f02b8a74f68c606680e4ae5f118b0dabad9" exitCode=0
Nov 01 22:28:38.824578 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.824549    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97" event={"ID":"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb","Type":"ContainerDied","Data":"54076969c4a228b6f70f7f93d67e0f02b8a74f68c606680e4ae5f118b0dabad9"}
Nov 01 22:28:38.830186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.830158    4531 generic.go:356] "Generic (PLEG): container finished" podID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerID="d4fe2c27daa9ff8f44ce02c625b129e78d041fbf0188c88b969732337ecdb7ce" exitCode=0
Nov 01 22:28:38.830321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.830268    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr" event={"ID":"bbeb220f-1e86-4dbf-b29e-841368dc64c3","Type":"ContainerDied","Data":"d4fe2c27daa9ff8f44ce02c625b129e78d041fbf0188c88b969732337ecdb7ce"}
Nov 01 22:28:38.832910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.832887    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk"
Nov 01 22:28:38.832910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.832903    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847as2rqk" event={"ID":"1eb3a3e5-aaee-4af7-bdf1-83050ca2b395","Type":"ContainerDied","Data":"04657db229db0d9689e87ddd7674da0d46a0821a0d6247a2ecbf3e94c9e7adea"}
Nov 01 22:28:38.833041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.832937    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04657db229db0d9689e87ddd7674da0d46a0821a0d6247a2ecbf3e94c9e7adea"
Nov 01 22:28:38.836662 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.836642    4531 generic.go:356] "Generic (PLEG): container finished" podID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerID="62d114a5bf53d53bf6fe9038bbe469e236d1d1d03d57cba8036105938f78fc4d" exitCode=0
Nov 01 22:28:38.836798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.836757    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff" event={"ID":"70c2e8cb-40ff-4488-bb99-87b7fdb9922c","Type":"ContainerDied","Data":"62d114a5bf53d53bf6fe9038bbe469e236d1d1d03d57cba8036105938f78fc4d"}
Nov 01 22:28:38.838770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.838748    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" event={"ID":"3aa762af-3281-4f40-bfb6-806acb624863","Type":"ContainerStarted","Data":"45f4182dda9155033226b7e8093d1bfc80537a7c84279fe74d063d321fdbe698"}
Nov 01 22:28:38.840555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.840528    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r" event={"ID":"dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc","Type":"ContainerDied","Data":"b335ca3ae6d3855d9ad1c6c24aa2ba91c44ac6e21aefccb095f0a1f646c4fb7a"}
Nov 01 22:28:38.840657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.840568    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847af9x7r"
Nov 01 22:28:38.840657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.840588    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b335ca3ae6d3855d9ad1c6c24aa2ba91c44ac6e21aefccb095f0a1f646c4fb7a"
Nov 01 22:28:38.975263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:38.975237    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:28:39.070481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.070456    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util\") pod \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") "
Nov 01 22:28:39.070580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.070497    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle\") pod \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") "
Nov 01 22:28:39.070580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.070520    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn\" (UniqueName: \"kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn\") pod \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\" (UID: \"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35\") "
Nov 01 22:28:39.070928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.070906    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle" pod "fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" (UID: "fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:39.072457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.072433    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn" pod "fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" (UID: "fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35"). InnerVolumeSpecName "kube-api-access-gtsgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:39.075535 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.075495    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util" pod "fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" (UID: "fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:39.077543 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.077503    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" podStartSLOduration=3.082285658 podStartE2EDuration="56.0774901s" totalImagesPullingTime="52.995204442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:27:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:39.076179023 +0000 UTC m=+300.393829042" watchObservedRunningTime="2026-11-01 22:28:39.0774901 +0000 UTC m=+300.395140100"
Nov 01 22:28:39.111059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.111032    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:28:39.113794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.113776    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:28:39.171921 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.171903    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util\") pod \"2f50cabc-7665-4200-9706-c857a6391b3c\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") "
Nov 01 22:28:39.172007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.171946    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle\") pod \"03722b82-e68a-4dda-9cea-aece7a654f72\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") "
Nov 01 22:28:39.172007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.171971    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle\") pod \"2f50cabc-7665-4200-9706-c857a6391b3c\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") "
Nov 01 22:28:39.172007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.171987    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util\") pod \"03722b82-e68a-4dda-9cea-aece7a654f72\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") "
Nov 01 22:28:39.172170 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172014    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk\" (UniqueName: \"kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk\") pod \"2f50cabc-7665-4200-9706-c857a6391b3c\" (UID: \"2f50cabc-7665-4200-9706-c857a6391b3c\") "
Nov 01 22:28:39.172170 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172072    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct\" (UniqueName: \"kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct\") pod \"03722b82-e68a-4dda-9cea-aece7a654f72\" (UID: \"03722b82-e68a-4dda-9cea-aece7a654f72\") "
Nov 01 22:28:39.172288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172270    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.172339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172294    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.172339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172309    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtsgn\" (UniqueName: \"kubernetes.io/projected/fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35-kube-api-access-gtsgn\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.172480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172460    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle" pod "2f50cabc-7665-4200-9706-c857a6391b3c" (UID: "2f50cabc-7665-4200-9706-c857a6391b3c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:39.172527 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.172489    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle" pod "03722b82-e68a-4dda-9cea-aece7a654f72" (UID: "03722b82-e68a-4dda-9cea-aece7a654f72"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:39.174436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.174415    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk" pod "2f50cabc-7665-4200-9706-c857a6391b3c" (UID: "2f50cabc-7665-4200-9706-c857a6391b3c"). InnerVolumeSpecName "kube-api-access-j5gsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:39.174436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.174426    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct" pod "03722b82-e68a-4dda-9cea-aece7a654f72" (UID: "03722b82-e68a-4dda-9cea-aece7a654f72"). InnerVolumeSpecName "kube-api-access-qm6ct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:39.178900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.178878    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util" pod "03722b82-e68a-4dda-9cea-aece7a654f72" (UID: "03722b82-e68a-4dda-9cea-aece7a654f72"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:39.178995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.178976    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util" pod "2f50cabc-7665-4200-9706-c857a6391b3c" (UID: "2f50cabc-7665-4200-9706-c857a6391b3c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:39.272798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.272778    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.272798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.272797    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.272899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.272805    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2f50cabc-7665-4200-9706-c857a6391b3c-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.272899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.272815    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03722b82-e68a-4dda-9cea-aece7a654f72-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.272899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.272823    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5gsk\" (UniqueName: \"kubernetes.io/projected/2f50cabc-7665-4200-9706-c857a6391b3c-kube-api-access-j5gsk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.272899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.272833    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm6ct\" (UniqueName: \"kubernetes.io/projected/03722b82-e68a-4dda-9cea-aece7a654f72-kube-api-access-qm6ct\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:39.850757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.850692    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx" event={"ID":"03722b82-e68a-4dda-9cea-aece7a654f72","Type":"ContainerDied","Data":"2c853062a1726959d3bdfe543037169a101efb2788995e70f724ade52d538f16"}
Nov 01 22:28:39.850757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.850764    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c853062a1726959d3bdfe543037169a101efb2788995e70f724ade52d538f16"
Nov 01 22:28:39.851323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.850704    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqjcwx"
Nov 01 22:28:39.853180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.853150    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4" event={"ID":"fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35","Type":"ContainerDied","Data":"25938861985d79a0140f367a75000b806dc753165bc4b70af076aef6227e7cf1"}
Nov 01 22:28:39.853322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.853197    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25938861985d79a0140f367a75000b806dc753165bc4b70af076aef6227e7cf1"
Nov 01 22:28:39.853322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.853163    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6942k4"
Nov 01 22:28:39.856015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.855982    4531 generic.go:356] "Generic (PLEG): container finished" podID="3aa762af-3281-4f40-bfb6-806acb624863" containerID="45f4182dda9155033226b7e8093d1bfc80537a7c84279fe74d063d321fdbe698" exitCode=0
Nov 01 22:28:39.856222 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.856201    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" event={"ID":"3aa762af-3281-4f40-bfb6-806acb624863","Type":"ContainerDied","Data":"45f4182dda9155033226b7e8093d1bfc80537a7c84279fe74d063d321fdbe698"}
Nov 01 22:28:39.867182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.867024    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg"
Nov 01 22:28:39.867182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.867070    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6c4dsg" event={"ID":"2f50cabc-7665-4200-9706-c857a6391b3c","Type":"ContainerDied","Data":"6a16a2c9ba33550adcb42b30c4963b50175c714991913581ae32f9ad0f704fc4"}
Nov 01 22:28:39.867182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.867108    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a16a2c9ba33550adcb42b30c4963b50175c714991913581ae32f9ad0f704fc4"
Nov 01 22:28:39.990954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:39.990928    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:28:40.050931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.050911    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:28:40.053857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.053841    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:28:40.079744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079710    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util\") pod \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") "
Nov 01 22:28:40.079826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079761    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle\") pod \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") "
Nov 01 22:28:40.079826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079803    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95\" (UniqueName: \"kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95\") pod \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\" (UID: \"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb\") "
Nov 01 22:28:40.079911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079842    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh\" (UniqueName: \"kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh\") pod \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") "
Nov 01 22:28:40.079911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079891    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util\") pod \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") "
Nov 01 22:28:40.079999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079924    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle\") pod \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") "
Nov 01 22:28:40.079999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079951    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle\") pod \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") "
Nov 01 22:28:40.079999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.079989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util\") pod \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\" (UID: \"bbeb220f-1e86-4dbf-b29e-841368dc64c3\") "
Nov 01 22:28:40.080157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.080016    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n\" (UniqueName: \"kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n\") pod \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\" (UID: \"70c2e8cb-40ff-4488-bb99-87b7fdb9922c\") "
Nov 01 22:28:40.080552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.080511    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle" pod "f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" (UID: "f471ae5d-f7e6-4fa0-8b22-e521a7c756cb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:40.080637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.080546    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle" pod "70c2e8cb-40ff-4488-bb99-87b7fdb9922c" (UID: "70c2e8cb-40ff-4488-bb99-87b7fdb9922c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:40.080695 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.080642    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle" pod "bbeb220f-1e86-4dbf-b29e-841368dc64c3" (UID: "bbeb220f-1e86-4dbf-b29e-841368dc64c3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:40.082207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.082185    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95" pod "f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" (UID: "f471ae5d-f7e6-4fa0-8b22-e521a7c756cb"). InnerVolumeSpecName "kube-api-access-q8v95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:40.082581 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.082557    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh" pod "bbeb220f-1e86-4dbf-b29e-841368dc64c3" (UID: "bbeb220f-1e86-4dbf-b29e-841368dc64c3"). InnerVolumeSpecName "kube-api-access-q4tnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:40.082685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.082618    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n" pod "70c2e8cb-40ff-4488-bb99-87b7fdb9922c" (UID: "70c2e8cb-40ff-4488-bb99-87b7fdb9922c"). InnerVolumeSpecName "kube-api-access-dzk9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:40.086316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.086296    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util" pod "f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" (UID: "f471ae5d-f7e6-4fa0-8b22-e521a7c756cb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:40.087447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.087426    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util" pod "70c2e8cb-40ff-4488-bb99-87b7fdb9922c" (UID: "70c2e8cb-40ff-4488-bb99-87b7fdb9922c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:40.087524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.087484    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util" pod "bbeb220f-1e86-4dbf-b29e-841368dc64c3" (UID: "bbeb220f-1e86-4dbf-b29e-841368dc64c3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:40.181102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181055    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181075    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzk9n\" (UniqueName: \"kubernetes.io/projected/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-kube-api-access-dzk9n\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181085    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181094    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181103    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8v95\" (UniqueName: \"kubernetes.io/projected/f471ae5d-f7e6-4fa0-8b22-e521a7c756cb-kube-api-access-q8v95\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181112    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4tnh\" (UniqueName: \"kubernetes.io/projected/bbeb220f-1e86-4dbf-b29e-841368dc64c3-kube-api-access-q4tnh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181120    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181128    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bbeb220f-1e86-4dbf-b29e-841368dc64c3-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.181297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.181135    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/70c2e8cb-40ff-4488-bb99-87b7fdb9922c-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:40.574922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.574886    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:40.771496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.771464    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:40.874635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.874599    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97" event={"ID":"f471ae5d-f7e6-4fa0-8b22-e521a7c756cb","Type":"ContainerDied","Data":"524a20d6fe817b3d532a1a378c033a29b73cce510de58bcf0b20b1b44eb3d61a"}
Nov 01 22:28:40.875050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.874651    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="524a20d6fe817b3d532a1a378c033a29b73cce510de58bcf0b20b1b44eb3d61a"
Nov 01 22:28:40.875050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.874610    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cljt97"
Nov 01 22:28:40.876767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.876745    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr"
Nov 01 22:28:40.876894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.876797    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7kxxfr" event={"ID":"bbeb220f-1e86-4dbf-b29e-841368dc64c3","Type":"ContainerDied","Data":"70d2f6677c081820a02a8fc5cc9843021cc59fbcc2d775a2655d317201fab55a"}
Nov 01 22:28:40.876894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.876842    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70d2f6677c081820a02a8fc5cc9843021cc59fbcc2d775a2655d317201fab55a"
Nov 01 22:28:40.878945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.878926    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff"
Nov 01 22:28:40.879030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.878920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hg7ff" event={"ID":"70c2e8cb-40ff-4488-bb99-87b7fdb9922c","Type":"ContainerDied","Data":"ce9588d11c2731a87a194d3233c254d17899f6162163baeaba256a24fa73e611"}
Nov 01 22:28:40.879030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.878976    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce9588d11c2731a87a194d3233c254d17899f6162163baeaba256a24fa73e611"
Nov 01 22:28:40.951254 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.951213    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:40.988247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:40.988225    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:28:41.088171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.088116    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle\") pod \"3aa762af-3281-4f40-bfb6-806acb624863\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") "
Nov 01 22:28:41.088283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.088185    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util\") pod \"3aa762af-3281-4f40-bfb6-806acb624863\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") "
Nov 01 22:28:41.088283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.088215    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds\" (UniqueName: \"kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds\") pod \"3aa762af-3281-4f40-bfb6-806acb624863\" (UID: \"3aa762af-3281-4f40-bfb6-806acb624863\") "
Nov 01 22:28:41.088506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.088485    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle" pod "3aa762af-3281-4f40-bfb6-806acb624863" (UID: "3aa762af-3281-4f40-bfb6-806acb624863"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:41.090201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.090182    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds" pod "3aa762af-3281-4f40-bfb6-806acb624863" (UID: "3aa762af-3281-4f40-bfb6-806acb624863"). InnerVolumeSpecName "kube-api-access-f4kds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:28:41.095284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.095260    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util" pod "3aa762af-3281-4f40-bfb6-806acb624863" (UID: "3aa762af-3281-4f40-bfb6-806acb624863"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:28:41.161297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.161267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:41.189610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.189583    4531 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-util\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:41.189610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.189606    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4kds\" (UniqueName: \"kubernetes.io/projected/3aa762af-3281-4f40-bfb6-806acb624863-kube-api-access-f4kds\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:41.189786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.189616    4531 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3aa762af-3281-4f40-bfb6-806acb624863-bundle\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:28:41.237651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.237625    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"]
Nov 01 22:28:41.238015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.237993    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="util"
Nov 01 22:28:41.238015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238009    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238022    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238027    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238035    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238039    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238049    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238053    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238066    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238071    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238076    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238081    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238087    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238091    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238098    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238102    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238109    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238113    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238118    4531 cpu_manager.go:414] "containerMap: removing container" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238122    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238128    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238132    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238142    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238147    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238153    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238157    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238163    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238167    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="extract"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238174    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238178    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238185    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238189    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="pull"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238199    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238204    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238209    4531 cpu_manager.go:414] "containerMap: removing container" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="util"
Nov 01 22:28:41.238198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238214    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238220    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238225    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238230    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238234    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238239    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238243    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238247    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238251    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238259    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238263    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238272    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238276    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238283    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238287    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238294    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238298    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238303    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238307    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238314    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238318    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238323    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238327    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238333    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238337    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238344    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238347    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238355    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238359    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238364    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238367    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="pull"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238373    4531 cpu_manager.go:414] "containerMap: removing container" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238377    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238384    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238388    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238393    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238397    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="util"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238480    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbeb220f-1e86-4dbf-b29e-841368dc64c3" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238493    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3ebe7b-dbe7-4f56-b95d-cad8d3ae8b35" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238500    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1eb3a3e5-aaee-4af7-bdf1-83050ca2b395" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238506    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70c2e8cb-40ff-4488-bb99-87b7fdb9922c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238513    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="03722b82-e68a-4dda-9cea-aece7a654f72" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238519    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f471ae5d-f7e6-4fa0-8b22-e521a7c756cb" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238525    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f682184-7ca2-47b5-ae61-d14198902cde" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238530    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd1c0771-c9eb-4322-aa79-a85ec4a3a4fc" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238536    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aee9ba54-701f-4559-be31-3e3df22f8c50" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238540    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aa762af-3281-4f40-bfb6-806acb624863" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238563    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a4ff41-bd6b-4386-8bdf-fb356e84909c" containerName="extract"
Nov 01 22:28:41.239390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.238572    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f50cabc-7665-4200-9706-c857a6391b3c" containerName="extract"
Nov 01 22:28:41.240849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.240834    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.290470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.290442    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9bc3a1f-b806-455f-a629-cf8ecc52b946-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.290588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.290487    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.290640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.290596    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9bc3a1f-b806-455f-a629-cf8ecc52b946-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.290640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.290634    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.290744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.290663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75dgd\" (UniqueName: \"kubernetes.io/projected/e9bc3a1f-b806-455f-a629-cf8ecc52b946-kube-api-access-75dgd\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.290821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.290803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.297866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.297844    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 22:28:41.297954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.297932    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 22:28:41.298028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.297994    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 22:28:41.298094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.298083    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 22:28:41.298127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.298118    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 22:28:41.298160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.298130    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jfzq8\""
Nov 01 22:28:41.391344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.391344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391321    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9bc3a1f-b806-455f-a629-cf8ecc52b946-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.391488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391347    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.391488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391382    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75dgd\" (UniqueName: \"kubernetes.io/projected/e9bc3a1f-b806-455f-a629-cf8ecc52b946-kube-api-access-75dgd\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.391488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391433    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.391641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9bc3a1f-b806-455f-a629-cf8ecc52b946-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.391773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.391750    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e9bc3a1f-b806-455f-a629-cf8ecc52b946-manifests\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.393812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.393785    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e9bc3a1f-b806-455f-a629-cf8ecc52b946-tmp\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.393904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.393841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-metrics-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.393904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.393857    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-webhook-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.394010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.393929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e9bc3a1f-b806-455f-a629-cf8ecc52b946-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.454556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.454538    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75dgd\" (UniqueName: \"kubernetes.io/projected/e9bc3a1f-b806-455f-a629-cf8ecc52b946-kube-api-access-75dgd\") pod \"opendatahub-operator-controller-manager-57f65ddd4-hfclr\" (UID: \"e9bc3a1f-b806-455f-a629-cf8ecc52b946\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.471951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.471929    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:41.493661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.493637    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"]
Nov 01 22:28:41.553665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.553647    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:41.669152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.669087    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:41.855940 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.855914    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:41.885849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.885825    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x"
Nov 01 22:28:41.885849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.885830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e6446rj8x" event={"ID":"3aa762af-3281-4f40-bfb6-806acb624863","Type":"ContainerDied","Data":"6efc6689945f1ce596f7272d951e8aa0e5d0dc204b65b5e65f7577251fa14f63"}
Nov 01 22:28:41.886219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:41.885884    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6efc6689945f1ce596f7272d951e8aa0e5d0dc204b65b5e65f7577251fa14f63"
Nov 01 22:28:42.060020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:42.059984    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"]
Nov 01 22:28:42.065966 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:28:42.065938    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bc3a1f_b806_455f_a629_cf8ecc52b946.slice/crio-49e6382305fe67cac4dd00580e0bada3223dd5a65ca144be9b3f5de631a647c5 WatchSource:0}: Error finding container 49e6382305fe67cac4dd00580e0bada3223dd5a65ca144be9b3f5de631a647c5: Status 404 returned error can't find the container with id 49e6382305fe67cac4dd00580e0bada3223dd5a65ca144be9b3f5de631a647c5
Nov 01 22:28:42.067652 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:42.067634    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:28:42.145051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:42.145028    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:42.383172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:42.383082    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:42.524033 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:28:42.523997    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:28:42.891925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:42.891896    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr" event={"ID":"e9bc3a1f-b806-455f-a629-cf8ecc52b946","Type":"ContainerStarted","Data":"49e6382305fe67cac4dd00580e0bada3223dd5a65ca144be9b3f5de631a647c5"}
Nov 01 22:28:43.952715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:43.952679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:45.245521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:45.245491    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:45.712348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:45.712319    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:46.913922 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:46.913838    4531 generic.go:356] "Generic (PLEG): container finished" podID="e9bc3a1f-b806-455f-a629-cf8ecc52b946" containerID="d7c9d8551bc1b81d37fa032677b66a7552df1b89b1e3d5c166e21e0c86398c60" exitCode=0
Nov 01 22:28:46.914273 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:46.913934    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr" event={"ID":"e9bc3a1f-b806-455f-a629-cf8ecc52b946","Type":"ContainerDied","Data":"d7c9d8551bc1b81d37fa032677b66a7552df1b89b1e3d5c166e21e0c86398c60"}
Nov 01 22:28:48.821281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.821246    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"]
Nov 01 22:28:48.827099 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.827077    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:48.837712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.837684    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"]
Nov 01 22:28:48.880884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.880862    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 01 22:28:48.880998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.880886    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 01 22:28:48.880998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.880902    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-42xxl\""
Nov 01 22:28:48.925864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.925836    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr" event={"ID":"e9bc3a1f-b806-455f-a629-cf8ecc52b946","Type":"ContainerStarted","Data":"96b467f209495d108dc53820aaba68b97bd64c944f6d9757cdbcbc397cbb0cb2"}
Nov 01 22:28:48.925968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.925954    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:48.951243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.951220    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr5nm\" (UniqueName: \"kubernetes.io/projected/b2208d8c-ee53-4f43-905e-79240b3939f3-kube-api-access-qr5nm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh\" (UID: \"b2208d8c-ee53-4f43-905e-79240b3939f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:48.951338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:48.951262    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2208d8c-ee53-4f43-905e-79240b3939f3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh\" (UID: \"b2208d8c-ee53-4f43-905e-79240b3939f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:49.052362 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.052341    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr5nm\" (UniqueName: \"kubernetes.io/projected/b2208d8c-ee53-4f43-905e-79240b3939f3-kube-api-access-qr5nm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh\" (UID: \"b2208d8c-ee53-4f43-905e-79240b3939f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:49.052444 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.052382    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2208d8c-ee53-4f43-905e-79240b3939f3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh\" (UID: \"b2208d8c-ee53-4f43-905e-79240b3939f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:49.052782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.052754    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2208d8c-ee53-4f43-905e-79240b3939f3-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh\" (UID: \"b2208d8c-ee53-4f43-905e-79240b3939f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:49.128275 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.128211    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr5nm\" (UniqueName: \"kubernetes.io/projected/b2208d8c-ee53-4f43-905e-79240b3939f3-kube-api-access-qr5nm\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh\" (UID: \"b2208d8c-ee53-4f43-905e-79240b3939f3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:49.138978 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.138960    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"
Nov 01 22:28:49.150640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.150605    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr" podStartSLOduration=2.753292298 podStartE2EDuration="8.150591544s" totalImagesPullingTime="4.979655012s" totalInitContainerRuntime="417.644234ms" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:49.147370574 +0000 UTC m=+310.465020586" watchObservedRunningTime="2026-11-01 22:28:49.150591544 +0000 UTC m=+310.468241545"
Nov 01 22:28:49.254105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.254062    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:49.436992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.436879    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:49.570214 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.570187    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh"]
Nov 01 22:28:49.575869 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:28:49.575841    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2208d8c_ee53_4f43_905e_79240b3939f3.slice/crio-8f5ef71cdddc086b20d72901fa89a91263addcddfa5878a29cc975adcba91c51 WatchSource:0}: Error finding container 8f5ef71cdddc086b20d72901fa89a91263addcddfa5878a29cc975adcba91c51: Status 404 returned error can't find the container with id 8f5ef71cdddc086b20d72901fa89a91263addcddfa5878a29cc975adcba91c51
Nov 01 22:28:49.931962 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:49.931933    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh" event={"ID":"b2208d8c-ee53-4f43-905e-79240b3939f3","Type":"ContainerStarted","Data":"8f5ef71cdddc086b20d72901fa89a91263addcddfa5878a29cc975adcba91c51"}
Nov 01 22:28:51.538553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:51.538517    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:52.711467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:52.711437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:52.949475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:52.949442    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh" event={"ID":"b2208d8c-ee53-4f43-905e-79240b3939f3","Type":"ContainerStarted","Data":"e2214bef4a5bbdd2adc945223413a18cc032866bcccc4a6e0a886d9931160db2"}
Nov 01 22:28:53.155345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:53.155297    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-jtdqh" podStartSLOduration=2.072022144 podStartE2EDuration="5.155284995s" totalImagesPullingTime="3.083262851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:28:53.153931594 +0000 UTC m=+314.471581594" watchObservedRunningTime="2026-11-01 22:28:53.155284995 +0000 UTC m=+314.472934996"
Nov 01 22:28:54.566675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:54.566644    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:55.287773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:55.287742    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:55.491905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:55.491875    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:56.742223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.742190    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"]
Nov 01 22:28:56.745923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.745902    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:28:56.796440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.796418    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wdxn8\""
Nov 01 22:28:56.819264 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.819241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbmk\" (UniqueName: \"kubernetes.io/projected/4a74e855-f374-4e3a-ac80-233426d448fa-kube-api-access-bdbmk\") pod \"authorino-operator-7b7bb4bc77-ptxkz\" (UID: \"4a74e855-f374-4e3a-ac80-233426d448fa\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:28:56.842875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.842848    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 22:28:56.842992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.842872    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 22:28:56.845075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.845056    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"]
Nov 01 22:28:56.920656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.920629    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbmk\" (UniqueName: \"kubernetes.io/projected/4a74e855-f374-4e3a-ac80-233426d448fa-kube-api-access-bdbmk\") pod \"authorino-operator-7b7bb4bc77-ptxkz\" (UID: \"4a74e855-f374-4e3a-ac80-233426d448fa\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:28:56.981246 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:56.981222    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbmk\" (UniqueName: \"kubernetes.io/projected/4a74e855-f374-4e3a-ac80-233426d448fa-kube-api-access-bdbmk\") pod \"authorino-operator-7b7bb4bc77-ptxkz\" (UID: \"4a74e855-f374-4e3a-ac80-233426d448fa\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:28:57.058362 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:57.058292    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:28:57.430160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:57.430082    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"]
Nov 01 22:28:57.436186 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:28:57.436148    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a74e855_f374_4e3a_ac80_233426d448fa.slice/crio-57bbd43a8d6f4b5b520550ad3cf2257132b930d0c0b71e43e8ad1b010e3a5ee0 WatchSource:0}: Error finding container 57bbd43a8d6f4b5b520550ad3cf2257132b930d0c0b71e43e8ad1b010e3a5ee0: Status 404 returned error can't find the container with id 57bbd43a8d6f4b5b520550ad3cf2257132b930d0c0b71e43e8ad1b010e3a5ee0
Nov 01 22:28:57.671281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:57.671237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:57.975253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:57.975217    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz" event={"ID":"4a74e855-f374-4e3a-ac80-233426d448fa","Type":"ContainerStarted","Data":"57bbd43a8d6f4b5b520550ad3cf2257132b930d0c0b71e43e8ad1b010e3a5ee0"}
Nov 01 22:28:58.140948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:58.140911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:58.382811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:58.382708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:28:59.935088 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:59.935059    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57f65ddd4-hfclr"
Nov 01 22:28:59.969131 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:28:59.969099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:00.200235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.200200    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:00.493673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.493598    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"]
Nov 01 22:29:00.546792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.546757    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:00.554633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.554603    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"]
Nov 01 22:29:00.598048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.598007    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-pmbts\""
Nov 01 22:29:00.652956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.652924    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqgrl\" (UniqueName: \"kubernetes.io/projected/f8dc0e0e-3872-43b3-b0c5-69993fb224d9-kube-api-access-pqgrl\") pod \"limitador-operator-controller-manager-5778bb4449-tqmt8\" (UID: \"f8dc0e0e-3872-43b3-b0c5-69993fb224d9\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:00.753707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.753625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqgrl\" (UniqueName: \"kubernetes.io/projected/f8dc0e0e-3872-43b3-b0c5-69993fb224d9-kube-api-access-pqgrl\") pod \"limitador-operator-controller-manager-5778bb4449-tqmt8\" (UID: \"f8dc0e0e-3872-43b3-b0c5-69993fb224d9\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:00.808848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.808818    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqgrl\" (UniqueName: \"kubernetes.io/projected/f8dc0e0e-3872-43b3-b0c5-69993fb224d9-kube-api-access-pqgrl\") pod \"limitador-operator-controller-manager-5778bb4449-tqmt8\" (UID: \"f8dc0e0e-3872-43b3-b0c5-69993fb224d9\") " pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:00.919676 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:00.919643    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:01.327590 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:01.327521    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"]
Nov 01 22:29:01.331418 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:01.331388    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8dc0e0e_3872_43b3_b0c5_69993fb224d9.slice/crio-59fdc77290f7226ceeadee7bbb2c041a39418a6555c3997132c4a6119615ff75 WatchSource:0}: Error finding container 59fdc77290f7226ceeadee7bbb2c041a39418a6555c3997132c4a6119615ff75: Status 404 returned error can't find the container with id 59fdc77290f7226ceeadee7bbb2c041a39418a6555c3997132c4a6119615ff75
Nov 01 22:29:02.000286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:02.000257    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8" event={"ID":"f8dc0e0e-3872-43b3-b0c5-69993fb224d9","Type":"ContainerStarted","Data":"59fdc77290f7226ceeadee7bbb2c041a39418a6555c3997132c4a6119615ff75"}
Nov 01 22:29:02.104655 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:02.104617    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:02.364941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:02.364901    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:02.788920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:02.788882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:02.987560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:02.987530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:03.007202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:03.007163    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz" event={"ID":"4a74e855-f374-4e3a-ac80-233426d448fa","Type":"ContainerStarted","Data":"31e923662a5579a5a4de7632d1f8c62dc54a9273a9b31526d83e7dd791d32aba"}
Nov 01 22:29:03.007368 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:03.007308    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:29:03.119514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:03.119406    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz" podStartSLOduration=2.612081171 podStartE2EDuration="7.119391016s" totalImagesPullingTime="4.507309845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:28:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:03.117780049 +0000 UTC m=+324.435430052" watchObservedRunningTime="2026-11-01 22:29:03.119391016 +0000 UTC m=+324.437041063"
Nov 01 22:29:04.702543 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:04.702508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:04.883253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:04.883221    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:05.019097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.019011    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8" event={"ID":"f8dc0e0e-3872-43b3-b0c5-69993fb224d9","Type":"ContainerStarted","Data":"4bb95bad84d36099356756e838f1ec8a20dbc0f9bdefdb2f19852fe58061c860"}
Nov 01 22:29:05.019246 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.019120    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:05.141917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.141866    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8" podStartSLOduration=2.1327453419999998 podStartE2EDuration="5.141852743s" totalImagesPullingTime="3.009107401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:05.141507338 +0000 UTC m=+326.459157363" watchObservedRunningTime="2026-11-01 22:29:05.141852743 +0000 UTC m=+326.459502744"
Nov 01 22:29:05.188541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.188506    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"]
Nov 01 22:29:05.191243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.191217    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:05.243637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.243611    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-b77h9\""
Nov 01 22:29:05.243808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.243615    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 01 22:29:05.256433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.256401    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"]
Nov 01 22:29:05.279440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.279372    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:05.296517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.296481    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rftld\" (UniqueName: \"kubernetes.io/projected/d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e-kube-api-access-rftld\") pod \"dns-operator-controller-manager-8557cbdb84-rjmf9\" (UID: \"d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:05.397600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.397564    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rftld\" (UniqueName: \"kubernetes.io/projected/d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e-kube-api-access-rftld\") pod \"dns-operator-controller-manager-8557cbdb84-rjmf9\" (UID: \"d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:05.461503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.461465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rftld\" (UniqueName: \"kubernetes.io/projected/d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e-kube-api-access-rftld\") pod \"dns-operator-controller-manager-8557cbdb84-rjmf9\" (UID: \"d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:05.503090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.503055    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:05.854288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.854215    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rvnl7"]
Nov 01 22:29:05.857667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.857647    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:05.871005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.870984    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"]
Nov 01 22:29:05.879577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.879545    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:05.880836 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:05.880810    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f01ba5_cb85_4957_b96b_d46eb9ef0d4e.slice/crio-89d3628e429838ab609b838a6dfca63ce3a7b5faf8c047e8b19ca231e1009b56 WatchSource:0}: Error finding container 89d3628e429838ab609b838a6dfca63ce3a7b5faf8c047e8b19ca231e1009b56: Status 404 returned error can't find the container with id 89d3628e429838ab609b838a6dfca63ce3a7b5faf8c047e8b19ca231e1009b56
Nov 01 22:29:05.913198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.913171    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qhr8d\""
Nov 01 22:29:05.913198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.913189    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 22:29:05.914047 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.914023    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rvnl7"]
Nov 01 22:29:05.914907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:05.914893    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 22:29:06.003289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.003261    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/550b2b06-8dad-4bb6-9dae-a0fe18dcde85-kube-api-access-zztpz\") pod \"cert-manager-597b95cb84-rvnl7\" (UID: \"550b2b06-8dad-4bb6-9dae-a0fe18dcde85\") " pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.003416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.003299    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/550b2b06-8dad-4bb6-9dae-a0fe18dcde85-bound-sa-token\") pod \"cert-manager-597b95cb84-rvnl7\" (UID: \"550b2b06-8dad-4bb6-9dae-a0fe18dcde85\") " pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.028948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.028918    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9" event={"ID":"d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e","Type":"ContainerStarted","Data":"89d3628e429838ab609b838a6dfca63ce3a7b5faf8c047e8b19ca231e1009b56"}
Nov 01 22:29:06.104365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.104303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/550b2b06-8dad-4bb6-9dae-a0fe18dcde85-kube-api-access-zztpz\") pod \"cert-manager-597b95cb84-rvnl7\" (UID: \"550b2b06-8dad-4bb6-9dae-a0fe18dcde85\") " pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.104365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.104341    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/550b2b06-8dad-4bb6-9dae-a0fe18dcde85-bound-sa-token\") pod \"cert-manager-597b95cb84-rvnl7\" (UID: \"550b2b06-8dad-4bb6-9dae-a0fe18dcde85\") " pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.171055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.171021    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zztpz\" (UniqueName: \"kubernetes.io/projected/550b2b06-8dad-4bb6-9dae-a0fe18dcde85-kube-api-access-zztpz\") pod \"cert-manager-597b95cb84-rvnl7\" (UID: \"550b2b06-8dad-4bb6-9dae-a0fe18dcde85\") " pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.171338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.171319    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/550b2b06-8dad-4bb6-9dae-a0fe18dcde85-bound-sa-token\") pod \"cert-manager-597b95cb84-rvnl7\" (UID: \"550b2b06-8dad-4bb6-9dae-a0fe18dcde85\") " pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.183068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.183041    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rvnl7"
Nov 01 22:29:06.307165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.307131    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:06.553291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.553263    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rvnl7"]
Nov 01 22:29:06.558883 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:06.558859    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod550b2b06_8dad_4bb6_9dae_a0fe18dcde85.slice/crio-7bea9e3af42f658fcb83b33d2e27eaa02b5d41eea7a19ea185a469e42516711f WatchSource:0}: Error finding container 7bea9e3af42f658fcb83b33d2e27eaa02b5d41eea7a19ea185a469e42516711f: Status 404 returned error can't find the container with id 7bea9e3af42f658fcb83b33d2e27eaa02b5d41eea7a19ea185a469e42516711f
Nov 01 22:29:06.954475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:06.954441    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:07.035860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:07.035828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rvnl7" event={"ID":"550b2b06-8dad-4bb6-9dae-a0fe18dcde85","Type":"ContainerStarted","Data":"7bea9e3af42f658fcb83b33d2e27eaa02b5d41eea7a19ea185a469e42516711f"}
Nov 01 22:29:07.157641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:07.157610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:07.981681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:07.981647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:08.202844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:08.202810    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:09.049441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:09.049408    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9" event={"ID":"d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e","Type":"ContainerStarted","Data":"e5c8621b052cbffa7ec8ce418d11ea58adf52816079d13b6dcbb3c6129481370"}
Nov 01 22:29:09.049828 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:09.049599    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:09.195566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:09.195504    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9" podStartSLOduration=1.17672967 podStartE2EDuration="4.195487544s" totalImagesPullingTime="3.018757874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:09.194094651 +0000 UTC m=+330.511744652" watchObservedRunningTime="2026-11-01 22:29:09.195487544 +0000 UTC m=+330.513137546"
Nov 01 22:29:11.062298 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:11.062261    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rvnl7" event={"ID":"550b2b06-8dad-4bb6-9dae-a0fe18dcde85","Type":"ContainerStarted","Data":"414e03fac2afcf2b1160007507c6592c476dd658edcc4acb4835b08df9a37ca4"}
Nov 01 22:29:11.169829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:11.169773    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rvnl7" podStartSLOduration=2.216528464 podStartE2EDuration="6.169755237s" totalImagesPullingTime="3.953226773s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:11.168692045 +0000 UTC m=+332.486342046" watchObservedRunningTime="2026-11-01 22:29:11.169755237 +0000 UTC m=+332.487405242"
Nov 01 22:29:12.329423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.329389    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"]
Nov 01 22:29:12.331904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.331886    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.428503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.428476    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fbpnd\""
Nov 01 22:29:12.430952 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.430931    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"]
Nov 01 22:29:12.461059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.461031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tcx88\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.461223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.461104    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczhb\" (UniqueName: \"kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb\") pod \"kuadrant-operator-controller-manager-678d87766f-tcx88\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.526839 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:29:12.526807    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:29:12.562171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.562144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tcx88\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.562318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.562194    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczhb\" (UniqueName: \"kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb\") pod \"kuadrant-operator-controller-manager-678d87766f-tcx88\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.562555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.562537    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-tcx88\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.615351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.615300    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczhb\" (UniqueName: \"kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb\") pod \"kuadrant-operator-controller-manager-678d87766f-tcx88\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:12.644121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:12.644100    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:13.141760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:13.141740    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"]
Nov 01 22:29:13.144471 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:13.144450    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85553d96_8d53_4f99_9b37_e5ada18e126e.slice/crio-a2117886ea086d85ab1107b773a1daadf9efa7b64f634a998668bb726d9f5551 WatchSource:0}: Error finding container a2117886ea086d85ab1107b773a1daadf9efa7b64f634a998668bb726d9f5551: Status 404 returned error can't find the container with id a2117886ea086d85ab1107b773a1daadf9efa7b64f634a998668bb726d9f5551
Nov 01 22:29:14.014828 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:14.014797    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ptxkz"
Nov 01 22:29:14.078040 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:14.078008    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88" event={"ID":"85553d96-8d53-4f99-9b37-e5ada18e126e","Type":"ContainerStarted","Data":"a2117886ea086d85ab1107b773a1daadf9efa7b64f634a998668bb726d9f5551"}
Nov 01 22:29:15.314513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:15.314479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:15.517870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:15.517839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:15.922954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:15.922918    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:16.031646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:16.031617    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-5778bb4449-tqmt8"
Nov 01 22:29:16.108625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:16.108595    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:16.520319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:16.520280    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:16.737504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:16.737474    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:17.095584 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:17.095546    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:18.623683 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:18.623626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:18.847346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:18.847306    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:19.019319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.019288    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:19.208447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.208421    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"]
Nov 01 22:29:19.234191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.234164    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.270956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.270215    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"]
Nov 01 22:29:19.313670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.313645    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-jnggr\""
Nov 01 22:29:19.353364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.353324    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:19.414548 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414525    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.414689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414565    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe145b4b-7752-4d48-806b-424e365ad4af-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.414689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414584    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe145b4b-7752-4d48-806b-424e365ad4af-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.414689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414646    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.414873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414741    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.414873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414781    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.414873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414797    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe145b4b-7752-4d48-806b-424e365ad4af-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.415038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414889    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg4dp\" (UniqueName: \"kubernetes.io/projected/fe145b4b-7752-4d48-806b-424e365ad4af-kube-api-access-cg4dp\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.415038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.414936    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516090    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516134    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe145b4b-7752-4d48-806b-424e365ad4af-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516158    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe145b4b-7752-4d48-806b-424e365ad4af-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516190    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516229    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516533    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516646    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516688    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516722    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516755    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe145b4b-7752-4d48-806b-424e365ad4af-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.516887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.516817    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg4dp\" (UniqueName: \"kubernetes.io/projected/fe145b4b-7752-4d48-806b-424e365ad4af-kube-api-access-cg4dp\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.517050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.517026    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.517117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.517096    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe145b4b-7752-4d48-806b-424e365ad4af-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.518493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.518469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe145b4b-7752-4d48-806b-424e365ad4af-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.519347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.519325    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe145b4b-7752-4d48-806b-424e365ad4af-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.533536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.533485    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:19.569562 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.569535    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe145b4b-7752-4d48-806b-424e365ad4af-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.569785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.569769    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg4dp\" (UniqueName: \"kubernetes.io/projected/fe145b4b-7752-4d48-806b-424e365ad4af-kube-api-access-cg4dp\") pod \"data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g\" (UID: \"fe145b4b-7752-4d48-806b-424e365ad4af\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.866887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.866796    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:19.924477 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:19.924344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:20.058070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.058042    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-rjmf9"
Nov 01 22:29:20.106945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.106915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:20.113986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.113959    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88" event={"ID":"85553d96-8d53-4f99-9b37-e5ada18e126e","Type":"ContainerStarted","Data":"927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6"}
Nov 01 22:29:20.114108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.114028    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:20.282578 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.282519    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88" podStartSLOduration=1.949638701 podStartE2EDuration="8.28250436s" totalImagesPullingTime="6.332865659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:20.279657896 +0000 UTC m=+341.597307897" watchObservedRunningTime="2026-11-01 22:29:20.28250436 +0000 UTC m=+341.600154360"
Nov 01 22:29:20.337722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.337689    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"]
Nov 01 22:29:20.343476 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:20.343442    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe145b4b_7752_4d48_806b_424e365ad4af.slice/crio-e50884c3dc987e8030c52014e5023fa8818a2f0592e369787618d1cef989e899 WatchSource:0}: Error finding container e50884c3dc987e8030c52014e5023fa8818a2f0592e369787618d1cef989e899: Status 404 returned error can't find the container with id e50884c3dc987e8030c52014e5023fa8818a2f0592e369787618d1cef989e899
Nov 01 22:29:20.346263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.346231    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:29:20.346337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.346320    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:29:20.346388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:20.346355    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:29:21.120812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.120771    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g" event={"ID":"fe145b4b-7752-4d48-806b-424e365ad4af","Type":"ContainerStarted","Data":"8b3638c2193ddc13cdabd3bde6db81d91f24947da211bda7f3797b114bde1b6b"}
Nov 01 22:29:21.121303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.120842    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g" event={"ID":"fe145b4b-7752-4d48-806b-424e365ad4af","Type":"ContainerStarted","Data":"e50884c3dc987e8030c52014e5023fa8818a2f0592e369787618d1cef989e899"}
Nov 01 22:29:21.236317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.236272    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g" podStartSLOduration=2.236257941 podStartE2EDuration="2.236257941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:21.234350374 +0000 UTC m=+342.552000372" watchObservedRunningTime="2026-11-01 22:29:21.236257941 +0000 UTC m=+342.553907991"
Nov 01 22:29:21.511166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.511130    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:21.687399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.687369    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:21.867467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.867400    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:21.871619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:21.871593    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:22.126742 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:22.126660    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:22.127596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:22.127571    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g"
Nov 01 22:29:23.398276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:23.398204    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:23.660906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:23.660834    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:23.838381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:23.838354    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:24.573616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.573584    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jqgx2"]
Nov 01 22:29:24.576979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.576958    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:24.627095 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.627077    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d2rr4\""
Nov 01 22:29:24.629836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.629812    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jqgx2"]
Nov 01 22:29:24.663368 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.663345    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzmsf\" (UniqueName: \"kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf\") pod \"authorino-57f7f8659f-jqgx2\" (UID: \"2e525058-260f-41f7-92d0-44ea4b4e985c\") " pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:24.763741 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.763702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:24.763874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.763853    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmsf\" (UniqueName: \"kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf\") pod \"authorino-57f7f8659f-jqgx2\" (UID: \"2e525058-260f-41f7-92d0-44ea4b4e985c\") " pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:24.818090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.818064    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmsf\" (UniqueName: \"kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf\") pod \"authorino-57f7f8659f-jqgx2\" (UID: \"2e525058-260f-41f7-92d0-44ea4b4e985c\") " pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:24.889695 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.889645    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:24.941717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:24.941510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:25.254734 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:25.254692    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jqgx2"]
Nov 01 22:29:25.260189 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:25.260159    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2e525058_260f_41f7_92d0_44ea4b4e985c.slice/crio-66c3bcb03df891dd77385cb66b447b5dd9f5a48768303ce50838c3753303c2a3 WatchSource:0}: Error finding container 66c3bcb03df891dd77385cb66b447b5dd9f5a48768303ce50838c3753303c2a3: Status 404 returned error can't find the container with id 66c3bcb03df891dd77385cb66b447b5dd9f5a48768303ce50838c3753303c2a3
Nov 01 22:29:26.149707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:26.149674    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jqgx2" event={"ID":"2e525058-260f-41f7-92d0-44ea4b4e985c","Type":"ContainerStarted","Data":"66c3bcb03df891dd77385cb66b447b5dd9f5a48768303ce50838c3753303c2a3"}
Nov 01 22:29:26.174478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:26.174452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:27.502670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.502609    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mp594"]
Nov 01 22:29:27.506577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.506551    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:29:27.556982 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.556953    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mp594"]
Nov 01 22:29:27.589033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.589002    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8btlh\" (UniqueName: \"kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh\") pod \"authorino-56c6c9ddd5-mp594\" (UID: \"b30906de-5db6-44f1-b5eb-93088ff0b865\") " pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:29:27.681646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.681601    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:27.689800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.689769    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8btlh\" (UniqueName: \"kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh\") pod \"authorino-56c6c9ddd5-mp594\" (UID: \"b30906de-5db6-44f1-b5eb-93088ff0b865\") " pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:29:27.745876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.745845    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8btlh\" (UniqueName: \"kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh\") pod \"authorino-56c6c9ddd5-mp594\" (UID: \"b30906de-5db6-44f1-b5eb-93088ff0b865\") " pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:29:27.820331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.820273    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:29:27.873164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:27.873143    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:28.049462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:28.049426    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:28.192382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:28.192300    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mp594"]
Nov 01 22:29:28.196229 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:28.196201    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb30906de_5db6_44f1_b5eb_93088ff0b865.slice/crio-8ee380924f849cbcaab1b63aba51db24672e5f660f73d9d427929b57ad5b2014 WatchSource:0}: Error finding container 8ee380924f849cbcaab1b63aba51db24672e5f660f73d9d427929b57ad5b2014: Status 404 returned error can't find the container with id 8ee380924f849cbcaab1b63aba51db24672e5f660f73d9d427929b57ad5b2014
Nov 01 22:29:29.167558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:29.167521    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mp594" event={"ID":"b30906de-5db6-44f1-b5eb-93088ff0b865","Type":"ContainerStarted","Data":"8ee380924f849cbcaab1b63aba51db24672e5f660f73d9d427929b57ad5b2014"}
Nov 01 22:29:29.169021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:29.168993    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jqgx2" event={"ID":"2e525058-260f-41f7-92d0-44ea4b4e985c","Type":"ContainerStarted","Data":"4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc"}
Nov 01 22:29:29.260932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:29.260901    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:29.274265 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:29.274222    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-jqgx2" podStartSLOduration=1.973560464 podStartE2EDuration="5.274209658s" totalImagesPullingTime="3.300649194s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:29.272826709 +0000 UTC m=+350.590476710" watchObservedRunningTime="2026-11-01 22:29:29.274209658 +0000 UTC m=+350.591859658"
Nov 01 22:29:29.445932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:29.445904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:30.175612 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:30.175577    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mp594" event={"ID":"b30906de-5db6-44f1-b5eb-93088ff0b865","Type":"ContainerStarted","Data":"97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e"}
Nov 01 22:29:30.283714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:30.283664    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-mp594" podStartSLOduration=1.98326208 podStartE2EDuration="3.283650817s" totalImagesPullingTime="1.300388737s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:30.281872665 +0000 UTC m=+351.599522666" watchObservedRunningTime="2026-11-01 22:29:30.283650817 +0000 UTC m=+351.601300817"
Nov 01 22:29:30.331717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:30.331680    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jqgx2"]
Nov 01 22:29:31.123948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:31.123920    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 22:29:31.181081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:31.181050    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-jqgx2" podUID="2e525058-260f-41f7-92d0-44ea4b4e985c" containerName="authorino" containerID="cri-o://4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc" gracePeriod=30
Nov 01 22:29:31.436146 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:31.436117    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:31.520801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:31.520769    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf\" (UniqueName: \"kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf\") pod \"2e525058-260f-41f7-92d0-44ea4b4e985c\" (UID: \"2e525058-260f-41f7-92d0-44ea4b4e985c\") "
Nov 01 22:29:31.522825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:31.522799    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf" pod "2e525058-260f-41f7-92d0-44ea4b4e985c" (UID: "2e525058-260f-41f7-92d0-44ea4b4e985c"). InnerVolumeSpecName "kube-api-access-xzmsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:29:31.622215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:31.622185    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzmsf\" (UniqueName: \"kubernetes.io/projected/2e525058-260f-41f7-92d0-44ea4b4e985c-kube-api-access-xzmsf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:29:32.187457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.187424    4531 generic.go:356] "Generic (PLEG): container finished" podID="2e525058-260f-41f7-92d0-44ea4b4e985c" containerID="4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc" exitCode=0
Nov 01 22:29:32.187919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.187466    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-jqgx2"
Nov 01 22:29:32.187919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.187510    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jqgx2" event={"ID":"2e525058-260f-41f7-92d0-44ea4b4e985c","Type":"ContainerDied","Data":"4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc"}
Nov 01 22:29:32.187919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.187558    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-jqgx2" event={"ID":"2e525058-260f-41f7-92d0-44ea4b4e985c","Type":"ContainerDied","Data":"66c3bcb03df891dd77385cb66b447b5dd9f5a48768303ce50838c3753303c2a3"}
Nov 01 22:29:32.187919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.187581    4531 scope.go:122] "RemoveContainer" containerID="4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc"
Nov 01 22:29:32.196967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.196948    4531 scope.go:122] "RemoveContainer" containerID="4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc"
Nov 01 22:29:32.197229 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:29:32.197209    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc\": container with ID starting with 4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc not found: ID does not exist" containerID="4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc"
Nov 01 22:29:32.197305 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.197238    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc"} err="failed to get container status \"4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc\": rpc error: code = NotFound desc = could not find container \"4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc\": container with ID starting with 4516a90de9bbeff31704a7560279a4e661be2316b616564d2c8ab7ef0afb60dc not found: ID does not exist"
Nov 01 22:29:32.346654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.346623    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jqgx2"]
Nov 01 22:29:32.350208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.350186    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-jqgx2"]
Nov 01 22:29:32.814486 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:32.814459    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e525058-260f-41f7-92d0-44ea4b4e985c" path="/var/lib/kubelet/pods/2e525058-260f-41f7-92d0-44ea4b4e985c/volumes"
Nov 01 22:29:35.693686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:35.693656    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:35.866665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:35.866635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:36.296226 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:36.296193    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:36.474339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:36.474307    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:37.975827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:37.975796    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:38.194635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:38.194603    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:39.705525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:39.705490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:39.904587 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:39.904557    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:42.097700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:42.097644    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:42.335102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:42.335066    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:42.533266 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:29:42.533227    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:29:44.909198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.909169    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"]
Nov 01 22:29:44.909763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.909633    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2e525058-260f-41f7-92d0-44ea4b4e985c" containerName="authorino"
Nov 01 22:29:44.909763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.909648    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e525058-260f-41f7-92d0-44ea4b4e985c" containerName="authorino"
Nov 01 22:29:44.909763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.909756    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e525058-260f-41f7-92d0-44ea4b4e985c" containerName="authorino"
Nov 01 22:29:44.916984 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.916964    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:44.918605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.918578    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:44.966153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.966123    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 22:29:44.966252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.966152    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 22:29:44.966252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:44.966250    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-kgs49\""
Nov 01 22:29:45.031588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.031558    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/f1d29b80-c299-450d-9e81-8bca05696da9-kube-api-access-btql4\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.031713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.031598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1d29b80-c299-450d-9e81-8bca05696da9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.031713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.031617    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1d29b80-c299-450d-9e81-8bca05696da9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.031713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.031632    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1d29b80-c299-450d-9e81-8bca05696da9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.099244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.099213    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:45.106395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.106377    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\""
Nov 01 22:29:45.108752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.108720    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"]
Nov 01 22:29:45.132660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.132638    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/f1d29b80-c299-450d-9e81-8bca05696da9-kube-api-access-btql4\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.132792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.132674    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1d29b80-c299-450d-9e81-8bca05696da9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.132792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.132693    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1d29b80-c299-450d-9e81-8bca05696da9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.132792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.132707    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1d29b80-c299-450d-9e81-8bca05696da9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.133055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.133036    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f1d29b80-c299-450d-9e81-8bca05696da9-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.133124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.133070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f1d29b80-c299-450d-9e81-8bca05696da9-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.133438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.133420    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f1d29b80-c299-450d-9e81-8bca05696da9-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.186256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.186204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btql4\" (UniqueName: \"kubernetes.io/projected/f1d29b80-c299-450d-9e81-8bca05696da9-kube-api-access-btql4\") pod \"kserve-module-controller-manager-7566cf5fc8-znqzz\" (UID: \"f1d29b80-c299-450d-9e81-8bca05696da9\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.230147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.230125    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:45.279671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.279214    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:45.655479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:45.654945    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"]
Nov 01 22:29:45.660848 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:45.660817    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d29b80_c299_450d_9e81_8bca05696da9.slice/crio-cd2f86203f4692a3423bb8173d6c01f5625dd59a10281bd3a69b367721e2e999 WatchSource:0}: Error finding container cd2f86203f4692a3423bb8173d6c01f5625dd59a10281bd3a69b367721e2e999: Status 404 returned error can't find the container with id cd2f86203f4692a3423bb8173d6c01f5625dd59a10281bd3a69b367721e2e999
Nov 01 22:29:46.263931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:46.263898    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz" event={"ID":"f1d29b80-c299-450d-9e81-8bca05696da9","Type":"ContainerStarted","Data":"cd2f86203f4692a3423bb8173d6c01f5625dd59a10281bd3a69b367721e2e999"}
Nov 01 22:29:47.900598 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:47.900526    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:48.079484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:48.079417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:49.069197 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.069170    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:49.278292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.278237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:49.283504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.283479    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz" event={"ID":"f1d29b80-c299-450d-9e81-8bca05696da9","Type":"ContainerStarted","Data":"4c0038c6d8a1d6b1fbc4184d0e04978cc2c34d160815b669e34ae48d05a75df5"}
Nov 01 22:29:49.283596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.283540    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:29:49.440821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.440746    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz" podStartSLOduration=2.043638233 podStartE2EDuration="5.440713874s" totalImagesPullingTime="3.397075641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:49.439040418 +0000 UTC m=+370.756690421" watchObservedRunningTime="2026-11-01 22:29:49.440713874 +0000 UTC m=+370.758363897"
Nov 01 22:29:49.674191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.674162    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"]
Nov 01 22:29:49.676862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.676843    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.731132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.731109    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Nov 01 22:29:49.731266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.731115    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 22:29:49.731345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.731287    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 22:29:49.731345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.731312    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8ks56\""
Nov 01 22:29:49.734045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.734011    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"]
Nov 01 22:29:49.774098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.774075    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/734d44d5-7f85-4dee-a5ee-5edae6fbfd54-data\") pod \"postgres-7cdbf6dc44-glt96\" (UID: \"734d44d5-7f85-4dee-a5ee-5edae6fbfd54\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.774190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.774143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbgjw\" (UniqueName: \"kubernetes.io/projected/734d44d5-7f85-4dee-a5ee-5edae6fbfd54-kube-api-access-wbgjw\") pod \"postgres-7cdbf6dc44-glt96\" (UID: \"734d44d5-7f85-4dee-a5ee-5edae6fbfd54\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.798011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.797946    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:49.875052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.875020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/734d44d5-7f85-4dee-a5ee-5edae6fbfd54-data\") pod \"postgres-7cdbf6dc44-glt96\" (UID: \"734d44d5-7f85-4dee-a5ee-5edae6fbfd54\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.875166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.875091    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbgjw\" (UniqueName: \"kubernetes.io/projected/734d44d5-7f85-4dee-a5ee-5edae6fbfd54-kube-api-access-wbgjw\") pod \"postgres-7cdbf6dc44-glt96\" (UID: \"734d44d5-7f85-4dee-a5ee-5edae6fbfd54\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.875389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.875370    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/734d44d5-7f85-4dee-a5ee-5edae6fbfd54-data\") pod \"postgres-7cdbf6dc44-glt96\" (UID: \"734d44d5-7f85-4dee-a5ee-5edae6fbfd54\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.928435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.928408    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbgjw\" (UniqueName: \"kubernetes.io/projected/734d44d5-7f85-4dee-a5ee-5edae6fbfd54-kube-api-access-wbgjw\") pod \"postgres-7cdbf6dc44-glt96\" (UID: \"734d44d5-7f85-4dee-a5ee-5edae6fbfd54\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:49.988261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:49.988208    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:50.359098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:50.358981    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"]
Nov 01 22:29:50.365029 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:29:50.364996    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734d44d5_7f85_4dee_a5ee_5edae6fbfd54.slice/crio-19fd3124f621bb16bc028e879ffa15c754faad29755e88ddfb6c1182190cd1cf WatchSource:0}: Error finding container 19fd3124f621bb16bc028e879ffa15c754faad29755e88ddfb6c1182190cd1cf: Status 404 returned error can't find the container with id 19fd3124f621bb16bc028e879ffa15c754faad29755e88ddfb6c1182190cd1cf
Nov 01 22:29:51.295292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:51.295258    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96" event={"ID":"734d44d5-7f85-4dee-a5ee-5edae6fbfd54","Type":"ContainerStarted","Data":"19fd3124f621bb16bc028e879ffa15c754faad29755e88ddfb6c1182190cd1cf"}
Nov 01 22:29:53.427871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:53.427831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:29:58.339918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:58.339847    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96" event={"ID":"734d44d5-7f85-4dee-a5ee-5edae6fbfd54","Type":"ContainerStarted","Data":"b9926f626c4e87961521709a99fa50a283ece1eeb09cd8cacb763fc4fcff6a6e"}
Nov 01 22:29:58.340296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:58.339970    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:29:58.448358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:29:58.448312    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96" podStartSLOduration=2.230558356 podStartE2EDuration="9.448299166s" totalImagesPullingTime="7.21774081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:29:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:29:58.446346287 +0000 UTC m=+379.763996288" watchObservedRunningTime="2026-11-01 22:29:58.448299166 +0000 UTC m=+379.765949185"
Nov 01 22:30:04.375392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:04.375362    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-glt96"
Nov 01 22:30:10.290147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:10.290118    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-znqzz"
Nov 01 22:30:11.349949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.349921    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-dgnrx"]
Nov 01 22:30:11.355640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.355614    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.405602 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.405572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:11.409380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.409357    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 01 22:30:11.409497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.409423    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 01 22:30:11.409563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.409496    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 01 22:30:11.409563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.409519    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-bkm5p\""
Nov 01 22:30:11.458502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.458475    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-dgnrx"]
Nov 01 22:30:11.460260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.460240    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.460324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.460271    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.460324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.460300    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krnwq\" (UniqueName: \"kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.561673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.561641    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.561673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.561674    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.561931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.561700    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krnwq\" (UniqueName: \"kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.564084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.564062    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.564196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.564123    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.616098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.616047    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krnwq\" (UniqueName: \"kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq\") pod \"maas-controller-579cf7c6d-dgnrx\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") " pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.633832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.633809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:11.669036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.669012    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:11.785396 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.785372    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-5vq6r"]
Nov 01 22:30:11.790985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.790963    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:11.846634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.846607    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-5vq6r"]
Nov 01 22:30:11.848894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.848850    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:11.966609 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.966580    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:11.966771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.966749    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:11.966848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:11.966778    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds2lq\" (UniqueName: \"kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.038792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.038768    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-dgnrx"]
Nov 01 22:30:12.043474 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:30:12.043452    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e3d308_5332_4ee2_9bbf_d2e91a06b5cd.slice/crio-091507e90a3c2ca7b7dc480ee6cd4a0fc71201c82277e14802ee4d6cddf888c3 WatchSource:0}: Error finding container 091507e90a3c2ca7b7dc480ee6cd4a0fc71201c82277e14802ee4d6cddf888c3: Status 404 returned error can't find the container with id 091507e90a3c2ca7b7dc480ee6cd4a0fc71201c82277e14802ee4d6cddf888c3
Nov 01 22:30:12.067078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.067042    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.067158 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.067144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.067204 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.067172    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds2lq\" (UniqueName: \"kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.069448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.069428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.069523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.069429    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.122036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.122012    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds2lq\" (UniqueName: \"kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq\") pod \"maas-controller-5c9f79b66c-5vq6r\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") " pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.404840 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.404761    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:12.422443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.422407    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-dgnrx" event={"ID":"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd","Type":"ContainerStarted","Data":"091507e90a3c2ca7b7dc480ee6cd4a0fc71201c82277e14802ee4d6cddf888c3"}
Nov 01 22:30:12.538754 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:30:12.538703    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:30:12.786639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:12.786607    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-5vq6r"]
Nov 01 22:30:12.791517 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:30:12.791489    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb92f36_91c2_4163_9ffb_c665e3059bd9.slice/crio-345b32f7f3ed4e7600a301b296afc09fba16fe6fda557c229ed40c984e4540bc WatchSource:0}: Error finding container 345b32f7f3ed4e7600a301b296afc09fba16fe6fda557c229ed40c984e4540bc: Status 404 returned error can't find the container with id 345b32f7f3ed4e7600a301b296afc09fba16fe6fda557c229ed40c984e4540bc
Nov 01 22:30:13.429388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:13.429345    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" event={"ID":"7fb92f36-91c2-4163-9ffb-c665e3059bd9","Type":"ContainerStarted","Data":"345b32f7f3ed4e7600a301b296afc09fba16fe6fda557c229ed40c984e4540bc"}
Nov 01 22:30:16.272156 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.272130    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:16.453094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.453064    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-dgnrx" event={"ID":"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd","Type":"ContainerStarted","Data":"95b817555027dd3ee6f4e2beee1f6bc9d3165a72a0010104cd5d26eafb2e13c9"}
Nov 01 22:30:16.453262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.453246    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:16.454716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.454692    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" event={"ID":"7fb92f36-91c2-4163-9ffb-c665e3059bd9","Type":"ContainerStarted","Data":"d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"}
Nov 01 22:30:16.454837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.454754    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:16.519854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.519830    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:16.564377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.564304    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-dgnrx" podStartSLOduration=1.818549286 podStartE2EDuration="5.56429191s" totalImagesPullingTime="3.745742624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:16.561310647 +0000 UTC m=+397.878960648" watchObservedRunningTime="2026-11-01 22:30:16.56429191 +0000 UTC m=+397.881941911"
Nov 01 22:30:16.669566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:16.669525    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" podStartSLOduration=2.5236250890000003 podStartE2EDuration="5.669513915s" totalImagesPullingTime="3.145888826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:16.668519342 +0000 UTC m=+397.986169345" watchObservedRunningTime="2026-11-01 22:30:16.669513915 +0000 UTC m=+397.987163916"
Nov 01 22:30:18.946206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:18.946174    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"]
Nov 01 22:30:18.949991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:18.949971    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:18.999384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:18.999358    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 01 22:30:18.999384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:18.999366    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 01 22:30:18.999384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:18.999386    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 01 22:30:19.045833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.045808    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-rcwmz\""
Nov 01 22:30:19.050024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.050003    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"]
Nov 01 22:30:19.128736 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.128702    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.128843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.128795    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a515ca03-c82e-4b92-9103-57606ab3d027-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.128843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.128833    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6wnm\" (UniqueName: \"kubernetes.io/projected/a515ca03-c82e-4b92-9103-57606ab3d027-kube-api-access-k6wnm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.128914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.128899    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.135441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.135378    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:19.230189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.230128    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.230189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.230180    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a515ca03-c82e-4b92-9103-57606ab3d027-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.230330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.230213    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6wnm\" (UniqueName: \"kubernetes.io/projected/a515ca03-c82e-4b92-9103-57606ab3d027-kube-api-access-k6wnm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.230330 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:30:19.230228    4531 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Nov 01 22:30:19.230330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.230256    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.230330 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:30:19.230299    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-proxy-source podName:a515ca03-c82e-4b92-9103-57606ab3d027 nodeName:}" failed. No retries permitted until 2026-11-01 22:30:19.73027643 +0000 UTC m=+401.047926425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-cl6j4" (UID: "a515ca03-c82e-4b92-9103-57606ab3d027") : configmap "usage-tenancy-proxy-source" not found
Nov 01 22:30:19.230845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.230827    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.232634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.232615    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a515ca03-c82e-4b92-9103-57606ab3d027-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.284387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.284362    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6wnm\" (UniqueName: \"kubernetes.io/projected/a515ca03-c82e-4b92-9103-57606ab3d027-kube-api-access-k6wnm\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.341697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.341674    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:19.735340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.735310    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.736080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.736061    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/a515ca03-c82e-4b92-9103-57606ab3d027-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-cl6j4\" (UID: \"a515ca03-c82e-4b92-9103-57606ab3d027\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:19.862837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:19.862809    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:20.233288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:20.233260    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"]
Nov 01 22:30:20.239048 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:30:20.239021    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda515ca03_c82e_4b92_9103_57606ab3d027.slice/crio-1a4ba3868d144e67121bb1afc5473bb58d51ba9ca549788bf0742768f05873c7 WatchSource:0}: Error finding container 1a4ba3868d144e67121bb1afc5473bb58d51ba9ca549788bf0742768f05873c7: Status 404 returned error can't find the container with id 1a4ba3868d144e67121bb1afc5473bb58d51ba9ca549788bf0742768f05873c7
Nov 01 22:30:20.486028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:20.485942    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4" event={"ID":"a515ca03-c82e-4b92-9103-57606ab3d027","Type":"ContainerStarted","Data":"1a4ba3868d144e67121bb1afc5473bb58d51ba9ca549788bf0742768f05873c7"}
Nov 01 22:30:27.469120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:27.469090    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:27.469572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:27.469178    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:30:27.716633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:27.716597    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-dgnrx"]
Nov 01 22:30:27.716886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:27.716860    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-dgnrx" podUID="88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" containerName="manager" containerID="cri-o://95b817555027dd3ee6f4e2beee1f6bc9d3165a72a0010104cd5d26eafb2e13c9" gracePeriod=30
Nov 01 22:30:28.541511 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.541475    4531 generic.go:356] "Generic (PLEG): container finished" podID="88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" containerID="95b817555027dd3ee6f4e2beee1f6bc9d3165a72a0010104cd5d26eafb2e13c9" exitCode=0
Nov 01 22:30:28.541895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.541572    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-dgnrx" event={"ID":"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd","Type":"ContainerDied","Data":"95b817555027dd3ee6f4e2beee1f6bc9d3165a72a0010104cd5d26eafb2e13c9"}
Nov 01 22:30:28.541895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.541676    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-dgnrx" event={"ID":"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd","Type":"ContainerDied","Data":"091507e90a3c2ca7b7dc480ee6cd4a0fc71201c82277e14802ee4d6cddf888c3"}
Nov 01 22:30:28.541895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.541701    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="091507e90a3c2ca7b7dc480ee6cd4a0fc71201c82277e14802ee4d6cddf888c3"
Nov 01 22:30:28.624847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.624817    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:28.725709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.725676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert\") pod \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") "
Nov 01 22:30:28.725911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.725773    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs\") pod \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") "
Nov 01 22:30:28.725911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.725830    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq\" (UniqueName: \"kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq\") pod \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\" (UID: \"88e3d308-5332-4ee2-9bbf-d2e91a06b5cd\") "
Nov 01 22:30:28.729013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.728981    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert" pod "88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" (UID: "88e3d308-5332-4ee2-9bbf-d2e91a06b5cd"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:28.729168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.729124    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq" pod "88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" (UID: "88e3d308-5332-4ee2-9bbf-d2e91a06b5cd"). InnerVolumeSpecName "kube-api-access-krnwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:30:28.729438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.729412    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs" pod "88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" (UID: "88e3d308-5332-4ee2-9bbf-d2e91a06b5cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:30:28.826995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.826962    4531 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-webhook-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:28.826995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.826996    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:28.827204 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:28.827013    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krnwq\" (UniqueName: \"kubernetes.io/projected/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd-kube-api-access-krnwq\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:30:29.548456 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:29.548424    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-dgnrx"
Nov 01 22:30:29.626904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:29.626869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:29.707350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:29.707313    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-dgnrx"]
Nov 01 22:30:29.753609 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:29.753578    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-dgnrx"]
Nov 01 22:30:30.816333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:30.816305    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" path="/var/lib/kubelet/pods/88e3d308-5332-4ee2-9bbf-d2e91a06b5cd/volumes"
Nov 01 22:30:35.593288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:35.593246    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4" event={"ID":"a515ca03-c82e-4b92-9103-57606ab3d027","Type":"ContainerStarted","Data":"d382b634b7378dc36d7ebdb3a7e0a408732898f3ce0508ce39a8ac873a061bfe"}
Nov 01 22:30:35.593791 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:35.593313    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:35.705349 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:35.705306    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4" podStartSLOduration=2.803000422 podStartE2EDuration="17.70529209s" totalImagesPullingTime="14.902291668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:35.70477775 +0000 UTC m=+417.022427750" watchObservedRunningTime="2026-11-01 22:30:35.70529209 +0000 UTC m=+417.022942092"
Nov 01 22:30:42.527210 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:30:42.527167    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:30:46.304238 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:46.304202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:46.604464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:46.604377    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-cl6j4"
Nov 01 22:30:46.660595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:46.660560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:48.567958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:48.567925    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:48.778155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:48.778121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:49.066180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.066149    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:49.571645 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.571614    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"]
Nov 01 22:30:49.572155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.572132    4531 cpu_manager.go:414] "containerMap: removing container" podUID="88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" containerName="manager"
Nov 01 22:30:49.572155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.572155    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" containerName="manager"
Nov 01 22:30:49.572346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.572259    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="88e3d308-5332-4ee2-9bbf-d2e91a06b5cd" containerName="manager"
Nov 01 22:30:49.580523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.580503    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.633367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.633342    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"]
Nov 01 22:30:49.676821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.676795    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 22:30:49.676821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.676807    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 22:30:49.676821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.676795    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 22:30:49.730995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.730970    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.731124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.731005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfl96\" (UniqueName: \"kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.731124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.731098    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.731198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.731138    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.832091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.832027    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.832091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.832066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfl96\" (UniqueName: \"kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.832266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.832103    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.832266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.832136    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.832703 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.832678    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.834638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.834613    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.834781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.834689    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.886092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.886064    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfl96\" (UniqueName: \"kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96\") pod \"maas-api-6d669c7fcd-tqp7x\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:49.894852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:49.894830    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:50.281108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:50.281073    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"]
Nov 01 22:30:50.286549 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:30:50.286518    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac816a54_8013_437a_96c9_4512f9601b92.slice/crio-9c7f56b4fd1e827b00644407633024948f6075fc360037141969942cd23b81ee WatchSource:0}: Error finding container 9c7f56b4fd1e827b00644407633024948f6075fc360037141969942cd23b81ee: Status 404 returned error can't find the container with id 9c7f56b4fd1e827b00644407633024948f6075fc360037141969942cd23b81ee
Nov 01 22:30:50.679594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:50.679516    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x" event={"ID":"ac816a54-8013-437a-96c9-4512f9601b92","Type":"ContainerStarted","Data":"9c7f56b4fd1e827b00644407633024948f6075fc360037141969942cd23b81ee"}
Nov 01 22:30:52.246780 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:52.246687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:52.455594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:52.455558    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:52.663469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:52.663392    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:52.909530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:52.909501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:53.087482 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.087449    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:53.273661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.273630    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:53.340824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.340745    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"]
Nov 01 22:30:53.345651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.345625    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.399332 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.399304    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"]
Nov 01 22:30:53.442028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.442003    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lc6q8\""
Nov 01 22:30:53.442028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.442004    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Nov 01 22:30:53.464195 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.464157    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zmqs\" (UniqueName: \"kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs\") pod \"llmisvc-controller-manager-869d5dcc5d-ndqf2\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.464366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.464218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ndqf2\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.470918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.470895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:53.565490 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.565462    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zmqs\" (UniqueName: \"kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs\") pod \"llmisvc-controller-manager-869d5dcc5d-ndqf2\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.565617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.565519    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ndqf2\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.568096 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.568070    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ndqf2\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.619196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.619139    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zmqs\" (UniqueName: \"kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs\") pod \"llmisvc-controller-manager-869d5dcc5d-ndqf2\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.663929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.663905    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:53.701099 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.701059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x" event={"ID":"ac816a54-8013-437a-96c9-4512f9601b92","Type":"ContainerStarted","Data":"89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b"}
Nov 01 22:30:53.701224 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.701203    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:53.811149 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:53.811105    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x" podStartSLOduration=2.3449686659999998 podStartE2EDuration="4.811088729s" totalImagesPullingTime="2.466120063s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:53.808382959 +0000 UTC m=+435.126032961" watchObservedRunningTime="2026-11-01 22:30:53.811088729 +0000 UTC m=+435.128738732"
Nov 01 22:30:54.037533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:54.037506    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"]
Nov 01 22:30:54.043184 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:30:54.043157    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9504ddcd_2fda_45ab_af53_2280972ec8a9.slice/crio-9157fb76d58083d2854d5c21dd06d8766715d4fd4dd59342be96652dfbabeb66 WatchSource:0}: Error finding container 9157fb76d58083d2854d5c21dd06d8766715d4fd4dd59342be96652dfbabeb66: Status 404 returned error can't find the container with id 9157fb76d58083d2854d5c21dd06d8766715d4fd4dd59342be96652dfbabeb66
Nov 01 22:30:54.536513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:54.536483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:54.709462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:54.709420    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2" event={"ID":"9504ddcd-2fda-45ab-af53-2280972ec8a9","Type":"ContainerStarted","Data":"9157fb76d58083d2854d5c21dd06d8766715d4fd4dd59342be96652dfbabeb66"}
Nov 01 22:30:57.729616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:57.729572    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2" event={"ID":"9504ddcd-2fda-45ab-af53-2280972ec8a9","Type":"ContainerStarted","Data":"614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7"}
Nov 01 22:30:57.730030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:57.729661    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:30:57.844867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:57.844821    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2" podStartSLOduration=1.84014791 podStartE2EDuration="4.844807903s" totalImagesPullingTime="3.004659993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:30:57.842703639 +0000 UTC m=+439.160353640" watchObservedRunningTime="2026-11-01 22:30:57.844807903 +0000 UTC m=+439.162457900"
Nov 01 22:30:59.340408 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:59.340374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:59.519639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:59.519599    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:59.695557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:59.695527    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:30:59.715356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:59.715328    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:30:59.891234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:30:59.891203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:31:00.074226 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.074144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:31:00.256747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.256680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:31:00.446572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.446540    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:31:00.551709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.551675    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w8m6l"]
Nov 01 22:31:00.564975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.564947    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:00.610304 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.610272    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w8m6l"]
Nov 01 22:31:00.635147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.635114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:31:00.738839 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.738763    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfhfg\" (UniqueName: \"kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg\") pod \"authorino-6f6944bcdd-w8m6l\" (UID: \"7d0803f1-829f-4d2f-b57e-157122656905\") " pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:00.846600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.842766    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfhfg\" (UniqueName: \"kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg\") pod \"authorino-6f6944bcdd-w8m6l\" (UID: \"7d0803f1-829f-4d2f-b57e-157122656905\") " pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:00.888337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.888298    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w8m6l"]
Nov 01 22:31:00.888843 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:31:00.888804    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-rfhfg], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6f6944bcdd-w8m6l" podUID="7d0803f1-829f-4d2f-b57e-157122656905"
Nov 01 22:31:00.896166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.896143    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfhfg\" (UniqueName: \"kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg\") pod \"authorino-6f6944bcdd-w8m6l\" (UID: \"7d0803f1-829f-4d2f-b57e-157122656905\") " pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:00.899771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.899707    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64c565bc9b-dnr58"]
Nov 01 22:31:00.975842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.975814    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64c565bc9b-dnr58"]
Nov 01 22:31:00.976000 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:00.975902    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:01.145182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.145156    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rkgl\" (UniqueName: \"kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl\") pod \"authorino-64c565bc9b-dnr58\" (UID: \"8f908b10-268a-4230-a14f-85a4e6b47509\") " pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:01.246020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.245985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rkgl\" (UniqueName: \"kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl\") pod \"authorino-64c565bc9b-dnr58\" (UID: \"8f908b10-268a-4230-a14f-85a4e6b47509\") " pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:01.260545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.260521    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64c565bc9b-dnr58"]
Nov 01 22:31:01.260811 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:31:01.260784    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4rkgl], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-64c565bc9b-dnr58" podUID="8f908b10-268a-4230-a14f-85a4e6b47509"
Nov 01 22:31:01.286125 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.286099    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d567bbd66-sqsvx"]
Nov 01 22:31:01.298919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.298851    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rkgl\" (UniqueName: \"kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl\") pod \"authorino-64c565bc9b-dnr58\" (UID: \"8f908b10-268a-4230-a14f-85a4e6b47509\") " pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:01.318287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.318262    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.342612 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.342541    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d567bbd66-sqsvx"]
Nov 01 22:31:01.366933 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.366908    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 01 22:31:01.447699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.447663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert\") pod \"authorino-d567bbd66-sqsvx\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") " pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.448161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.447810    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfvpm\" (UniqueName: \"kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm\") pod \"authorino-d567bbd66-sqsvx\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") " pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.548746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.548695    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert\") pod \"authorino-d567bbd66-sqsvx\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") " pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.548928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.548828    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfvpm\" (UniqueName: \"kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm\") pod \"authorino-d567bbd66-sqsvx\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") " pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.551426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.551397    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert\") pod \"authorino-d567bbd66-sqsvx\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") " pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.602706 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.602647    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfvpm\" (UniqueName: \"kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm\") pod \"authorino-d567bbd66-sqsvx\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") " pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.652629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.652593    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:31:01.757716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.757683    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:01.757897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.757688    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:01.787911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.787882    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:01.799998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.799974    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:01.955143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.955108    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl\" (UniqueName: \"kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl\") pod \"8f908b10-268a-4230-a14f-85a4e6b47509\" (UID: \"8f908b10-268a-4230-a14f-85a4e6b47509\") "
Nov 01 22:31:01.955323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.955195    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg\" (UniqueName: \"kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg\") pod \"7d0803f1-829f-4d2f-b57e-157122656905\" (UID: \"7d0803f1-829f-4d2f-b57e-157122656905\") "
Nov 01 22:31:01.957216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.957186    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg" pod "7d0803f1-829f-4d2f-b57e-157122656905" (UID: "7d0803f1-829f-4d2f-b57e-157122656905"). InnerVolumeSpecName "kube-api-access-rfhfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:01.957361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:01.957298    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl" pod "8f908b10-268a-4230-a14f-85a4e6b47509" (UID: "8f908b10-268a-4230-a14f-85a4e6b47509"). InnerVolumeSpecName "kube-api-access-4rkgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:02.051834 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.051800    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d567bbd66-sqsvx"]
Nov 01 22:31:02.056031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.056004    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rkgl\" (UniqueName: \"kubernetes.io/projected/8f908b10-268a-4230-a14f-85a4e6b47509-kube-api-access-4rkgl\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:02.056031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.056028    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfhfg\" (UniqueName: \"kubernetes.io/projected/7d0803f1-829f-4d2f-b57e-157122656905-kube-api-access-rfhfg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:02.057431 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:31:02.057398    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e4cabc_5a0b_4dce_ac15_73a2e5cfc06e.slice/crio-d8a4c44c9127854a432b224c4dd8ec2b0806f73e347a43277e4231b8d89493f8 WatchSource:0}: Error finding container d8a4c44c9127854a432b224c4dd8ec2b0806f73e347a43277e4231b8d89493f8: Status 404 returned error can't find the container with id d8a4c44c9127854a432b224c4dd8ec2b0806f73e347a43277e4231b8d89493f8
Nov 01 22:31:02.765920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.765881    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d567bbd66-sqsvx" event={"ID":"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e","Type":"ContainerStarted","Data":"d8a4c44c9127854a432b224c4dd8ec2b0806f73e347a43277e4231b8d89493f8"}
Nov 01 22:31:02.766357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.765942    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-w8m6l"
Nov 01 22:31:02.766357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.765953    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64c565bc9b-dnr58"
Nov 01 22:31:02.986053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.986015    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w8m6l"]
Nov 01 22:31:02.989241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:02.989214    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-w8m6l"]
Nov 01 22:31:03.147692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:03.147612    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-64c565bc9b-dnr58"]
Nov 01 22:31:03.150228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:03.150203    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-64c565bc9b-dnr58"]
Nov 01 22:31:04.924504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:04.781778    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d567bbd66-sqsvx" event={"ID":"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e","Type":"ContainerStarted","Data":"4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99"}
Nov 01 22:31:04.924504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:04.817374    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0803f1-829f-4d2f-b57e-157122656905" path="/var/lib/kubelet/pods/7d0803f1-829f-4d2f-b57e-157122656905/volumes"
Nov 01 22:31:04.924504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:04.817615    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f908b10-268a-4230-a14f-85a4e6b47509" path="/var/lib/kubelet/pods/8f908b10-268a-4230-a14f-85a4e6b47509/volumes"
Nov 01 22:31:04.924504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:04.888233    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-d567bbd66-sqsvx" podStartSLOduration=1.780992773 podStartE2EDuration="3.888213872s" totalImagesPullingTime="2.107221099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:31:04.887957634 +0000 UTC m=+446.205607635" watchObservedRunningTime="2026-11-01 22:31:04.888213872 +0000 UTC m=+446.205863874"
Nov 01 22:31:04.924504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:04.918301    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mp594"]
Nov 01 22:31:04.924504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:04.918481    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-mp594" podUID="b30906de-5db6-44f1-b5eb-93088ff0b865" containerName="authorino" containerID="cri-o://97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e" gracePeriod=30
Nov 01 22:31:05.238026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.238000    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:31:05.288165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.288135    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh\" (UniqueName: \"kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh\") pod \"b30906de-5db6-44f1-b5eb-93088ff0b865\" (UID: \"b30906de-5db6-44f1-b5eb-93088ff0b865\") "
Nov 01 22:31:05.290223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.290196    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh" pod "b30906de-5db6-44f1-b5eb-93088ff0b865" (UID: "b30906de-5db6-44f1-b5eb-93088ff0b865"). InnerVolumeSpecName "kube-api-access-8btlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:05.389757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.389671    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8btlh\" (UniqueName: \"kubernetes.io/projected/b30906de-5db6-44f1-b5eb-93088ff0b865-kube-api-access-8btlh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:05.788342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.788301    4531 generic.go:356] "Generic (PLEG): container finished" podID="b30906de-5db6-44f1-b5eb-93088ff0b865" containerID="97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e" exitCode=0
Nov 01 22:31:05.788526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.788351    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-mp594"
Nov 01 22:31:05.788526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.788389    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mp594" event={"ID":"b30906de-5db6-44f1-b5eb-93088ff0b865","Type":"ContainerDied","Data":"97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e"}
Nov 01 22:31:05.788526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.788435    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-mp594" event={"ID":"b30906de-5db6-44f1-b5eb-93088ff0b865","Type":"ContainerDied","Data":"8ee380924f849cbcaab1b63aba51db24672e5f660f73d9d427929b57ad5b2014"}
Nov 01 22:31:05.788526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.788457    4531 scope.go:122] "RemoveContainer" containerID="97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e"
Nov 01 22:31:05.790692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.790661    4531 generic.go:356] "Generic (PLEG): container finished" podID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerID="66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d" exitCode=0
Nov 01 22:31:05.790830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.790721    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" event={"ID":"747f48c3-52de-4ed9-a5b7-ea0d3960fd40","Type":"ContainerDied","Data":"66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"}
Nov 01 22:31:05.791259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.791236    4531 scope.go:122] "RemoveContainer" containerID="66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"
Nov 01 22:31:05.792474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.792446    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:31:05.792574 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.792557    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:31:05.792658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.792610    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:31:05.798148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.798133    4531 scope.go:122] "RemoveContainer" containerID="97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e"
Nov 01 22:31:05.798444 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:31:05.798427    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e\": container with ID starting with 97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e not found: ID does not exist" containerID="97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e"
Nov 01 22:31:05.798502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:05.798451    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e"} err="failed to get container status \"97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e\": rpc error: code = NotFound desc = could not find container \"97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e\": container with ID starting with 97f159497af47174e68c10aa33b62e41011a7bc32cd4a9710aa60dedf73a026e not found: ID does not exist"
Nov 01 22:31:06.052448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:06.052377    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mp594"]
Nov 01 22:31:06.056359 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:06.056330    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-mp594"]
Nov 01 22:31:06.800654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:06.800620    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" event={"ID":"747f48c3-52de-4ed9-a5b7-ea0d3960fd40","Type":"ContainerStarted","Data":"aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e"}
Nov 01 22:31:06.814693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:06.814661    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30906de-5db6-44f1-b5eb-93088ff0b865" path="/var/lib/kubelet/pods/b30906de-5db6-44f1-b5eb-93088ff0b865/volumes"
Nov 01 22:31:07.329358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:07.329325    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:31:07.330517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:07.330485    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dss7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 22:31:07.330632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:07.330536    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 22:31:08.273858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:08.273812    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:31:08.329467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:08.329421    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dss7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 22:31:08.329881 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:08.329485    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 22:31:09.329429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:09.329390    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dss7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 22:31:09.329621 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:09.329448    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 22:31:10.329465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:10.329433    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dss7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 22:31:10.329667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:10.329497    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 22:31:10.988794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:10.988754    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"]
Nov 01 22:31:10.989019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:10.988977    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x" podUID="ac816a54-8013-437a-96c9-4512f9601b92" containerName="maas-api" containerID="cri-o://89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b" gracePeriod=30
Nov 01 22:31:11.240485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.240430    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:31:11.328544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.328509    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-dss7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Nov 01 22:31:11.328672 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.328556    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Nov 01 22:31:11.347917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.347887    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls\") pod \"ac816a54-8013-437a-96c9-4512f9601b92\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") "
Nov 01 22:31:11.348303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.347934    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96\" (UniqueName: \"kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96\") pod \"ac816a54-8013-437a-96c9-4512f9601b92\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") "
Nov 01 22:31:11.348303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.347970    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca\") pod \"ac816a54-8013-437a-96c9-4512f9601b92\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") "
Nov 01 22:31:11.348303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.348082    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs\") pod \"ac816a54-8013-437a-96c9-4512f9601b92\" (UID: \"ac816a54-8013-437a-96c9-4512f9601b92\") "
Nov 01 22:31:11.348562 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.348525    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca" pod "ac816a54-8013-437a-96c9-4512f9601b92" (UID: "ac816a54-8013-437a-96c9-4512f9601b92"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:31:11.350455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.350414    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs" pod "ac816a54-8013-437a-96c9-4512f9601b92" (UID: "ac816a54-8013-437a-96c9-4512f9601b92"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:11.350628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.350605    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls" pod "ac816a54-8013-437a-96c9-4512f9601b92" (UID: "ac816a54-8013-437a-96c9-4512f9601b92"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:31:11.350816 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.350686    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96" pod "ac816a54-8013-437a-96c9-4512f9601b92" (UID: "ac816a54-8013-437a-96c9-4512f9601b92"). InnerVolumeSpecName "kube-api-access-cfl96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:31:11.449497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.449469    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:11.449497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.449495    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac816a54-8013-437a-96c9-4512f9601b92-maas-api-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:11.449670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.449509    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfl96\" (UniqueName: \"kubernetes.io/projected/ac816a54-8013-437a-96c9-4512f9601b92-kube-api-access-cfl96\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:11.449670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.449522    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac816a54-8013-437a-96c9-4512f9601b92-openshift-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:31:11.830863 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.830831    4531 generic.go:356] "Generic (PLEG): container finished" podID="ac816a54-8013-437a-96c9-4512f9601b92" containerID="89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b" exitCode=0
Nov 01 22:31:11.831039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.830884    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"
Nov 01 22:31:11.831039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.830907    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x" event={"ID":"ac816a54-8013-437a-96c9-4512f9601b92","Type":"ContainerDied","Data":"89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b"}
Nov 01 22:31:11.831039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.830955    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x" event={"ID":"ac816a54-8013-437a-96c9-4512f9601b92","Type":"ContainerDied","Data":"9c7f56b4fd1e827b00644407633024948f6075fc360037141969942cd23b81ee"}
Nov 01 22:31:11.831039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.830977    4531 scope.go:122] "RemoveContainer" containerID="89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b"
Nov 01 22:31:11.840866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.840849    4531 scope.go:122] "RemoveContainer" containerID="89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b"
Nov 01 22:31:11.841101 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:31:11.841082    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b\": container with ID starting with 89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b not found: ID does not exist" containerID="89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b"
Nov 01 22:31:11.841160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.841111    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b"} err="failed to get container status \"89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b\": rpc error: code = NotFound desc = could not find container \"89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b\": container with ID starting with 89280ce33ade12e30312de10feaa1c34166cff053f0f2f15df781bd14763f50b not found: ID does not exist"
Nov 01 22:31:11.984243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:11.984217    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"]
Nov 01 22:31:12.030686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:12.030662    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-tqp7x"]
Nov 01 22:31:12.332774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:12.332744    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:31:12.333630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:12.333612    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:31:12.525394 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:31:12.525365    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:31:12.815202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:12.815177    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac816a54-8013-437a-96c9-4512f9601b92" path="/var/lib/kubelet/pods/ac816a54-8013-437a-96c9-4512f9601b92/volumes"
Nov 01 22:31:28.737890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:28.737853    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:31:29.955630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:31:29.955596    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:31:42.524496 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:31:42.524456    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:32:06.327397 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327328    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"]
Nov 01 22:32:06.327954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327772    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ac816a54-8013-437a-96c9-4512f9601b92" containerName="maas-api"
Nov 01 22:32:06.327954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327786    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac816a54-8013-437a-96c9-4512f9601b92" containerName="maas-api"
Nov 01 22:32:06.327954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327820    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b30906de-5db6-44f1-b5eb-93088ff0b865" containerName="authorino"
Nov 01 22:32:06.327954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327825    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30906de-5db6-44f1-b5eb-93088ff0b865" containerName="authorino"
Nov 01 22:32:06.327954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327913    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30906de-5db6-44f1-b5eb-93088ff0b865" containerName="authorino"
Nov 01 22:32:06.327954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.327926    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac816a54-8013-437a-96c9-4512f9601b92" containerName="maas-api"
Nov 01 22:32:06.331414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.331396    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.385987 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.385960    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"]
Nov 01 22:32:06.405428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.405396    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khf9k\" (UniqueName: \"kubernetes.io/projected/dd9d1da5-3391-4195-b1b4-4a839c7232e6-kube-api-access-khf9k\") pod \"llmisvc-controller-manager-6466c554d8-qbc8z\" (UID: \"dd9d1da5-3391-4195-b1b4-4a839c7232e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.405561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.405546    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd9d1da5-3391-4195-b1b4-4a839c7232e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-qbc8z\" (UID: \"dd9d1da5-3391-4195-b1b4-4a839c7232e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.506633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.506608    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd9d1da5-3391-4195-b1b4-4a839c7232e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-qbc8z\" (UID: \"dd9d1da5-3391-4195-b1b4-4a839c7232e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.506832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.506659    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khf9k\" (UniqueName: \"kubernetes.io/projected/dd9d1da5-3391-4195-b1b4-4a839c7232e6-kube-api-access-khf9k\") pod \"llmisvc-controller-manager-6466c554d8-qbc8z\" (UID: \"dd9d1da5-3391-4195-b1b4-4a839c7232e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.509000 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.508979    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dd9d1da5-3391-4195-b1b4-4a839c7232e6-cert\") pod \"llmisvc-controller-manager-6466c554d8-qbc8z\" (UID: \"dd9d1da5-3391-4195-b1b4-4a839c7232e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.560167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.560146    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khf9k\" (UniqueName: \"kubernetes.io/projected/dd9d1da5-3391-4195-b1b4-4a839c7232e6-kube-api-access-khf9k\") pod \"llmisvc-controller-manager-6466c554d8-qbc8z\" (UID: \"dd9d1da5-3391-4195-b1b4-4a839c7232e6\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:06.644542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:06.644486    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:07.012019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:07.011988    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"]
Nov 01 22:32:07.017117 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:32:07.017087    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9d1da5_3391_4195_b1b4_4a839c7232e6.slice/crio-553824ac634d34558cfcea67b65858b6bde320c88113070b72b5bcda6e7a066c WatchSource:0}: Error finding container 553824ac634d34558cfcea67b65858b6bde320c88113070b72b5bcda6e7a066c: Status 404 returned error can't find the container with id 553824ac634d34558cfcea67b65858b6bde320c88113070b72b5bcda6e7a066c
Nov 01 22:32:07.125463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:07.125432    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z" event={"ID":"dd9d1da5-3391-4195-b1b4-4a839c7232e6","Type":"ContainerStarted","Data":"553824ac634d34558cfcea67b65858b6bde320c88113070b72b5bcda6e7a066c"}
Nov 01 22:32:08.134180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:08.134094    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z" event={"ID":"dd9d1da5-3391-4195-b1b4-4a839c7232e6","Type":"ContainerStarted","Data":"37c78c23508957e5249f3df1893323e0060c478ce7cb99ad9cac792825554600"}
Nov 01 22:32:08.134180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:08.134165    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:08.242460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:08.242407    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z" podStartSLOduration=1.506104662 podStartE2EDuration="2.242393886s" totalImagesPullingTime="736.289224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:32:08.239984579 +0000 UTC m=+509.557634577" watchObservedRunningTime="2026-11-01 22:32:08.242393886 +0000 UTC m=+509.560043886"
Nov 01 22:32:12.525163 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:32:12.525125    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:32:30.297561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:30.297532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:32:33.526980 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:33.526942    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:32:39.142002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.141971    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-qbc8z"
Nov 01 22:32:39.298013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.297978    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"]
Nov 01 22:32:39.298191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.298141    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2" podUID="9504ddcd-2fda-45ab-af53-2280972ec8a9" containerName="manager" containerID="cri-o://614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7" gracePeriod=30
Nov 01 22:32:39.537334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.537306    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:32:39.674150 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.674088    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert\" (UniqueName: \"kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert\") pod \"9504ddcd-2fda-45ab-af53-2280972ec8a9\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") "
Nov 01 22:32:39.674267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.674171    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs\" (UniqueName: \"kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs\") pod \"9504ddcd-2fda-45ab-af53-2280972ec8a9\" (UID: \"9504ddcd-2fda-45ab-af53-2280972ec8a9\") "
Nov 01 22:32:39.676105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.676076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert" pod "9504ddcd-2fda-45ab-af53-2280972ec8a9" (UID: "9504ddcd-2fda-45ab-af53-2280972ec8a9"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:32:39.676211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.676106    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs" pod "9504ddcd-2fda-45ab-af53-2280972ec8a9" (UID: "9504ddcd-2fda-45ab-af53-2280972ec8a9"). InnerVolumeSpecName "kube-api-access-8zmqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:32:39.775315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.775290    4531 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9504ddcd-2fda-45ab-af53-2280972ec8a9-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:39.775315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:39.775310    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zmqs\" (UniqueName: \"kubernetes.io/projected/9504ddcd-2fda-45ab-af53-2280972ec8a9-kube-api-access-8zmqs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:32:40.123379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.123348    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:32:40.314143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.314116    4531 generic.go:356] "Generic (PLEG): container finished" podID="9504ddcd-2fda-45ab-af53-2280972ec8a9" containerID="614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7" exitCode=0
Nov 01 22:32:40.314588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.314177    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"
Nov 01 22:32:40.314588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.314201    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2" event={"ID":"9504ddcd-2fda-45ab-af53-2280972ec8a9","Type":"ContainerDied","Data":"614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7"}
Nov 01 22:32:40.314588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.314275    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2" event={"ID":"9504ddcd-2fda-45ab-af53-2280972ec8a9","Type":"ContainerDied","Data":"9157fb76d58083d2854d5c21dd06d8766715d4fd4dd59342be96652dfbabeb66"}
Nov 01 22:32:40.314588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.314310    4531 scope.go:122] "RemoveContainer" containerID="614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7"
Nov 01 22:32:40.324336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.324318    4531 scope.go:122] "RemoveContainer" containerID="614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7"
Nov 01 22:32:40.324669 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:32:40.324651    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7\": container with ID starting with 614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7 not found: ID does not exist" containerID="614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7"
Nov 01 22:32:40.324755 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.324679    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7"} err="failed to get container status \"614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7\": rpc error: code = NotFound desc = could not find container \"614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7\": container with ID starting with 614d6037bc0e77049d39d2e053d95725ca18c9692b79271e8780d86f62d225c7 not found: ID does not exist"
Nov 01 22:32:40.332616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.332595    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:32:40.473272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.473249    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"]
Nov 01 22:32:40.522750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.520681    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ndqf2"]
Nov 01 22:32:40.814680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:40.814621    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9504ddcd-2fda-45ab-af53-2280972ec8a9" path="/var/lib/kubelet/pods/9504ddcd-2fda-45ab-af53-2280972ec8a9/volumes"
Nov 01 22:32:42.524717 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:32:42.524680    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:32:44.340850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:44.340763    4531 generic.go:356] "Generic (PLEG): container finished" podID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerID="d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943" exitCode=1
Nov 01 22:32:44.341308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:44.340843    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" event={"ID":"7fb92f36-91c2-4163-9ffb-c665e3059bd9","Type":"ContainerDied","Data":"d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"}
Nov 01 22:32:44.341308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:44.341232    4531 scope.go:122] "RemoveContainer" containerID="d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"
Nov 01 22:32:46.353637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:46.353597    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" event={"ID":"7fb92f36-91c2-4163-9ffb-c665e3059bd9","Type":"ContainerStarted","Data":"3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0"}
Nov 01 22:32:46.354091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:46.353888    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:32:57.368598 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:57.368570    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 22:32:58.996391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:32:58.996344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:03.095040 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:03.095005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:06.340624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:06.340574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:09.621576 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:09.621539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:12.528151 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:33:12.528114    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:33:14.017980 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:14.017938    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:20.923481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:20.923336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:24.287147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:24.287063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:27.329629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:27.329569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:30.636677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:30.636647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:34.998094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:34.998059    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:38.756398 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.756367    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"]
Nov 01 22:33:38.756875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.756826    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9504ddcd-2fda-45ab-af53-2280972ec8a9" containerName="manager"
Nov 01 22:33:38.756875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.756839    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9504ddcd-2fda-45ab-af53-2280972ec8a9" containerName="manager"
Nov 01 22:33:38.757006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.756991    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9504ddcd-2fda-45ab-af53-2280972ec8a9" containerName="manager"
Nov 01 22:33:38.760430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.760410    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.873868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.873839    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"]
Nov 01 22:33:38.898405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.898373    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.903365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.903334    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.903610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.903594    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rm4n\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.903915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.903889    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.904067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.903937    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.904067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.903977    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.904067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.903994    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.904236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.904222    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:38.904291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:38.904273    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005620    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005705    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005836    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005886    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.005994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005963    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rm4n\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.005993    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.006024    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.006034    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.006135    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.006295    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006486 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.006439    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.006578 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.006559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.008258 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.008239    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.008458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.008439    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.065454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.065429    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.066007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.065985    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rm4n\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n\") pod \"maas-default-gateway-openshift-default-7f694df8bc-tvx5d\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.078123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.078100    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:39.503135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.503096    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"]
Nov 01 22:33:39.508595 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:33:39.508569    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a94edad_54be_4e70_bfcb_8ba8c2b6374c.slice/crio-833f0f95a66f108ca00a1dc6df38d57f0d19291c462ffa281b7ae641ff549515 WatchSource:0}: Error finding container 833f0f95a66f108ca00a1dc6df38d57f0d19291c462ffa281b7ae641ff549515: Status 404 returned error can't find the container with id 833f0f95a66f108ca00a1dc6df38d57f0d19291c462ffa281b7ae641ff549515
Nov 01 22:33:39.510668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.510637    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:33:39.510759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.510701    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:33:39.510804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.510760    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:33:39.688098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.688062    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" event={"ID":"3a94edad-54be-4e70-bfcb-8ba8c2b6374c","Type":"ContainerStarted","Data":"4a9de364354d66c7710414134167b2bfda9c1cbbdd0e4f0e39f750c8e7cc10fe"}
Nov 01 22:33:39.688297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.688123    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" event={"ID":"3a94edad-54be-4e70-bfcb-8ba8c2b6374c","Type":"ContainerStarted","Data":"833f0f95a66f108ca00a1dc6df38d57f0d19291c462ffa281b7ae641ff549515"}
Nov 01 22:33:39.802345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:39.802237    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podStartSLOduration=1.8022200590000002 podStartE2EDuration="1.802220059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:39.799008279 +0000 UTC m=+601.116658279" watchObservedRunningTime="2026-11-01 22:33:39.802220059 +0000 UTC m=+601.119870061"
Nov 01 22:33:40.083807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:40.083513    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:40.083807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:40.083655    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:40.083807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:40.083701    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:41.080749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:41.080055    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:41.080749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:41.080112    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:42.082750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:42.082103    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:42.082750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:42.082162    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:42.525881 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:33:42.525846    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:33:43.081750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:43.081123    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:43.081750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:43.081189    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:44.085765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:44.078435    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:44.085765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:44.078494    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:45.082094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:45.082057    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:45.082485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:45.082457    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:46.079323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:46.079290    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Nov 01 22:33:46.079805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:46.079345    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Nov 01 22:33:47.490545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.490504    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f75d86897-txbsw"]
Nov 01 22:33:47.495831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.495806    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.548441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.548401    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f75d86897-txbsw"]
Nov 01 22:33:47.581011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.580970    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02c23e68-7466-4755-9a16-47f14ec6e7cd-tls-cert\") pod \"authorino-5f75d86897-txbsw\" (UID: \"02c23e68-7466-4755-9a16-47f14ec6e7cd\") " pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.581177 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.581025    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljcms\" (UniqueName: \"kubernetes.io/projected/02c23e68-7466-4755-9a16-47f14ec6e7cd-kube-api-access-ljcms\") pod \"authorino-5f75d86897-txbsw\" (UID: \"02c23e68-7466-4755-9a16-47f14ec6e7cd\") " pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.682649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.682610    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02c23e68-7466-4755-9a16-47f14ec6e7cd-tls-cert\") pod \"authorino-5f75d86897-txbsw\" (UID: \"02c23e68-7466-4755-9a16-47f14ec6e7cd\") " pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.682836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.682675    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljcms\" (UniqueName: \"kubernetes.io/projected/02c23e68-7466-4755-9a16-47f14ec6e7cd-kube-api-access-ljcms\") pod \"authorino-5f75d86897-txbsw\" (UID: \"02c23e68-7466-4755-9a16-47f14ec6e7cd\") " pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.685280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.685253    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/02c23e68-7466-4755-9a16-47f14ec6e7cd-tls-cert\") pod \"authorino-5f75d86897-txbsw\" (UID: \"02c23e68-7466-4755-9a16-47f14ec6e7cd\") " pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.730228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.730200    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:47.730520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.730500    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:47.731251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.731229    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 22:33:47.737122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.737102    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljcms\" (UniqueName: \"kubernetes.io/projected/02c23e68-7466-4755-9a16-47f14ec6e7cd-kube-api-access-ljcms\") pod \"authorino-5f75d86897-txbsw\" (UID: \"02c23e68-7466-4755-9a16-47f14ec6e7cd\") " pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.799484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.799420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:47.812506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.812483    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f75d86897-txbsw"
Nov 01 22:33:47.990366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:47.990330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:48.180295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:48.180226    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f75d86897-txbsw"]
Nov 01 22:33:48.185615 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:33:48.185576    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02c23e68_7466_4755_9a16_47f14ec6e7cd.slice/crio-f2a0c999ce2a8306e9a17edfef661458e6ebb530649b27f99750a15f7a96856b WatchSource:0}: Error finding container f2a0c999ce2a8306e9a17edfef661458e6ebb530649b27f99750a15f7a96856b: Status 404 returned error can't find the container with id f2a0c999ce2a8306e9a17edfef661458e6ebb530649b27f99750a15f7a96856b
Nov 01 22:33:48.187148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:48.187127    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:33:48.742054 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:48.742016    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f75d86897-txbsw" event={"ID":"02c23e68-7466-4755-9a16-47f14ec6e7cd","Type":"ContainerStarted","Data":"f2a0c999ce2a8306e9a17edfef661458e6ebb530649b27f99750a15f7a96856b"}
Nov 01 22:33:50.756320 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:50.756280    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f75d86897-txbsw" event={"ID":"02c23e68-7466-4755-9a16-47f14ec6e7cd","Type":"ContainerStarted","Data":"abcb4397784e9461a5ff0a1a1b13bcd9854e025b7fca8d2f35f39e790218349d"}
Nov 01 22:33:50.867158 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:50.867107    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f75d86897-txbsw" podStartSLOduration=2.115113482 podStartE2EDuration="3.86709205s" totalImagesPullingTime="1.751978568s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:33:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:33:50.864349999 +0000 UTC m=+612.181999999" watchObservedRunningTime="2026-11-01 22:33:50.86709205 +0000 UTC m=+612.184742114"
Nov 01 22:33:50.915667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:50.915629    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d567bbd66-sqsvx"]
Nov 01 22:33:50.915942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:50.915906    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-d567bbd66-sqsvx" podUID="91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" containerName="authorino" containerID="cri-o://4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99" gracePeriod=30
Nov 01 22:33:51.180147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.180119    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:33:51.221279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.221240    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert\" (UniqueName: \"kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert\") pod \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") "
Nov 01 22:33:51.221454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.221391    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm\" (UniqueName: \"kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm\") pod \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\" (UID: \"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e\") "
Nov 01 22:33:51.223534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.223498    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm" pod "91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" (UID: "91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e"). InnerVolumeSpecName "kube-api-access-nfvpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:33:51.236268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.236237    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert" pod "91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" (UID: "91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:33:51.323237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.323158    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfvpm\" (UniqueName: \"kubernetes.io/projected/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-kube-api-access-nfvpm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:33:51.323237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.323187    4531 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e-tls-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:33:51.765514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.765444    4531 generic.go:356] "Generic (PLEG): container finished" podID="91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" containerID="4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99" exitCode=0
Nov 01 22:33:51.765514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.765525    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d567bbd66-sqsvx" event={"ID":"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e","Type":"ContainerDied","Data":"4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99"}
Nov 01 22:33:51.766297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.765569    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d567bbd66-sqsvx" event={"ID":"91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e","Type":"ContainerDied","Data":"d8a4c44c9127854a432b224c4dd8ec2b0806f73e347a43277e4231b8d89493f8"}
Nov 01 22:33:51.766297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.765587    4531 scope.go:122] "RemoveContainer" containerID="4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99"
Nov 01 22:33:51.766297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.765488    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d567bbd66-sqsvx"
Nov 01 22:33:51.779046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.778750    4531 scope.go:122] "RemoveContainer" containerID="4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99"
Nov 01 22:33:51.779287 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:33:51.779240    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99\": container with ID starting with 4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99 not found: ID does not exist" containerID="4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99"
Nov 01 22:33:51.779402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.779283    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99"} err="failed to get container status \"4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99\": rpc error: code = NotFound desc = could not find container \"4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99\": container with ID starting with 4a4488e0742acd041206b468e07d37387cfa1368d8caf3326dd594448b9f1c99 not found: ID does not exist"
Nov 01 22:33:51.921639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.921596    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-d567bbd66-sqsvx"]
Nov 01 22:33:51.966178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:51.966146    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-d567bbd66-sqsvx"]
Nov 01 22:33:52.816405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:52.816371    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" path="/var/lib/kubelet/pods/91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e/volumes"
Nov 01 22:33:57.927142 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:57.927105    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:33:58.113926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:33:58.113899    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:03.109658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:03.109623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:12.524691 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:34:12.524650    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:34:22.876902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:22.876873    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:23.342279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:23.342252    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:24.081927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.081894    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"]
Nov 01 22:34:24.082372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.082358    4531 cpu_manager.go:414] "containerMap: removing container" podUID="91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" containerName="authorino"
Nov 01 22:34:24.082415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.082373    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" containerName="authorino"
Nov 01 22:34:24.082449 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.082430    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e4cabc-5a0b-4dce-ac15-73a2e5cfc06e" containerName="authorino"
Nov 01 22:34:24.085637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.085620    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214461    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214516    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214635    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214777 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214845    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mkdb\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214899    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.214994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.214934    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.215046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.215019    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.283361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.283335    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"]
Nov 01 22:34:24.315573 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315550    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315601    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mkdb\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315618    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315651    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315692    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315751    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315797    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.315873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.315842    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.316042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.316020    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.316116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.316094    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.316295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.316273    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.316373 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.316309    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.316494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.316476    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.318448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.318424    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.318548 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.318529    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.368775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.368705    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mkdb\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.369046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.369028    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-nhzgt\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.398046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.398024    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:24.823672 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.823638    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"]
Nov 01 22:34:24.828071 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:34:24.828044    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c9a213_b1b8_4ae5_8d4a_e15fa591732f.slice/crio-d5d83966844f5890283e78ea6001199d80f7590f092c0a56a90acc1bca1a789a WatchSource:0}: Error finding container d5d83966844f5890283e78ea6001199d80f7590f092c0a56a90acc1bca1a789a: Status 404 returned error can't find the container with id d5d83966844f5890283e78ea6001199d80f7590f092c0a56a90acc1bca1a789a
Nov 01 22:34:24.829912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.829884    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:34:24.829997 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.829946    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:34:24.829997 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.829976    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:34:24.863501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.863477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:24.963643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.963609    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" event={"ID":"17c9a213-b1b8-4ae5-8d4a-e15fa591732f","Type":"ContainerStarted","Data":"88d54acbd5573517c57ca5af555fdc2b10d1a8f5deece503eea4938d851f6585"}
Nov 01 22:34:24.963856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:24.963668    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" event={"ID":"17c9a213-b1b8-4ae5-8d4a-e15fa591732f","Type":"ContainerStarted","Data":"d5d83966844f5890283e78ea6001199d80f7590f092c0a56a90acc1bca1a789a"}
Nov 01 22:34:25.065549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:25.065521    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:25.081942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:25.081840    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podStartSLOduration=1.081824135 podStartE2EDuration="1.081824135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:34:25.08055677 +0000 UTC m=+646.398206803" watchObservedRunningTime="2026-11-01 22:34:25.081824135 +0000 UTC m=+646.399474138"
Nov 01 22:34:25.399303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:25.399217    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:25.400565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:25.400538    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:25.400682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:25.400582    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:26.324612 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:26.324572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:26.399389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:26.399350    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:26.399540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:26.399405    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:26.524166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:26.524136    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:27.398811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:27.398781    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:27.399207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:27.398830    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:28.398597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:28.398562    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:28.398804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:28.398616    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:29.399296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:29.399267    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:29.399668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:29.399344    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:30.399344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:30.399306    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:30.399855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:30.399376    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:31.052447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:31.052412    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:31.398620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:31.398541    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Nov 01 22:34:31.398620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:31.398602    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Nov 01 22:34:33.054533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:33.054496    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:33.055564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:33.055543    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:33.055986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:33.055967    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:34:42.525576 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:34:42.525534    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:34:52.528990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:52.528904    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:52.770646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:52.770617    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:53.321552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:53.321520    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:53.740776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:53.740718    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:54.102874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:54.102786    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:54.354147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:54.354049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:54.575566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:54.575534    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:54.781616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:54.781559    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:54.982651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:54.982614    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:55.177510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:55.177417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:55.382781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:55.382715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:55.608372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:55.608334    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:57.123249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:57.123220    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:57.601799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:57.601768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:58.416899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:58.416858    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:58.687850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:58.687769    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:59.738809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:59.738767    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:34:59.939354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:34:59.939321    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:00.790043 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:00.790012    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:01.045258 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.045190    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:01.271017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.270985    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:01.508083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.508042    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:01.527253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.527222    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"]
Nov 01 22:35:01.531006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.530985    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.585904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.585866    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-5b8fd2-dockercfg-lftsk\""
Nov 01 22:35:01.602292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.602258    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.602450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.602436    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkrk\" (UniqueName: \"kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk\") pod \"payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.632511 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.632478    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-5b8fd2\""
Nov 01 22:35:01.635460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.635428    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"]
Nov 01 22:35:01.664419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.664382    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"]
Nov 01 22:35:01.668185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.668163    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.703173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.703136    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkrk\" (UniqueName: \"kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk\") pod \"payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.703338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.703226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.704240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.704213    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.732611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.732572    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"]
Nov 01 22:35:01.739565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.739531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:01.758529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.758455    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkrk\" (UniqueName: \"kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk\") pod \"payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.804106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.804072    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwdpv\" (UniqueName: \"kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv\") pod \"payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.804513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.804197    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.845891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.845856    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:01.905442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.905410    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.905573 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.905552    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwdpv\" (UniqueName: \"kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv\") pod \"payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.906397 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.906371    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.963873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.963838    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwdpv\" (UniqueName: \"kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv\") pod \"payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:01.980677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:01.980646    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:02.162101 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:02.162069    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:02.226205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:02.226171    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"]
Nov 01 22:35:02.231208 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:02.231183    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac9bf77_abe7_4fd5_b7d5_7b1d0e406e21.slice/crio-436fd06577cb4cdce8af429fc76a835a59c8ee1f230ec1e74376f38908edeff6 WatchSource:0}: Error finding container 436fd06577cb4cdce8af429fc76a835a59c8ee1f230ec1e74376f38908edeff6: Status 404 returned error can't find the container with id 436fd06577cb4cdce8af429fc76a835a59c8ee1f230ec1e74376f38908edeff6
Nov 01 22:35:02.358225 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:02.358132    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"]
Nov 01 22:35:02.439966 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:02.439933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:03.199947 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:03.199913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" event={"ID":"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21","Type":"ContainerStarted","Data":"436fd06577cb4cdce8af429fc76a835a59c8ee1f230ec1e74376f38908edeff6"}
Nov 01 22:35:03.201153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:03.201129    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" event={"ID":"f33ff84c-4c87-4ec2-9a33-25cef07051a0","Type":"ContainerStarted","Data":"a4a9125328798e5a0478368b41e914a5543b053216171460eebecbce30de74a9"}
Nov 01 22:35:03.900698 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:03.900662    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:04.140083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:04.139988    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:04.441528 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:04.441496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:04.638807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:04.638776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:04.891183 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:04.891143    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:05.233957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:05.233920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:05.561234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:05.561167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:05.793856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:05.793824    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:06.226967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:06.226925    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" event={"ID":"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21","Type":"ContainerStarted","Data":"b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8"}
Nov 01 22:35:06.227230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:06.227022    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:06.228631 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:06.228600    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" event={"ID":"f33ff84c-4c87-4ec2-9a33-25cef07051a0","Type":"ContainerStarted","Data":"21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac"}
Nov 01 22:35:06.228769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:06.228666    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:06.336665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:06.336616    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" podStartSLOduration=2.241654101 podStartE2EDuration="5.336602706s" totalImagesPullingTime="3.094948605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:06.33482671 +0000 UTC m=+687.652476711" watchObservedRunningTime="2026-11-01 22:35:06.336602706 +0000 UTC m=+687.654252707"
Nov 01 22:35:06.444300 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:06.444261    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" podStartSLOduration=2.530366647 podStartE2EDuration="5.444250543s" totalImagesPullingTime="2.913883896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:06.442047455 +0000 UTC m=+687.759697457" watchObservedRunningTime="2026-11-01 22:35:06.444250543 +0000 UTC m=+687.761900545"
Nov 01 22:35:07.193977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:07.193946    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:07.387406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:07.387376    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:08.617771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:08.617734    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:08.922610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:08.922517    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:08.991331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:08.991301    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"]
Nov 01 22:35:08.995955 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:08.995934    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.049289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.049265    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-d6101b\""
Nov 01 22:35:09.095345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.095320    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-d6101b-dockercfg-8s5xx\""
Nov 01 22:35:09.099566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.099542    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"]
Nov 01 22:35:09.136889 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.136701    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"]
Nov 01 22:35:09.141127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.141102    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.178543 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.178472    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcjx\" (UniqueName: \"kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx\") pod \"payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.178671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.178650    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.192950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.192921    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"]
Nov 01 22:35:09.279637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.279607    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.279832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.279651    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4npl\" (UniqueName: \"kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl\") pod \"payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.279832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.279687    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcjx\" (UniqueName: \"kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx\") pod \"payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.279832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.279780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.280351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.280326    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.333811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.333781    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcjx\" (UniqueName: \"kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx\") pod \"payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.381067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.381043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.381171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.381136    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4npl\" (UniqueName: \"kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl\") pod \"payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.381663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.381645    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.435233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.435181    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4npl\" (UniqueName: \"kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl\") pod \"payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.457842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.457811    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:09.609285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.609250    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:09.747393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.747359    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:09.825141 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.825116    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"]
Nov 01 22:35:09.829745 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:09.829705    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2f1c24_981c_4224_834a_1169ee831c91.slice/crio-fe4c075c800bd00af123a6041d39e194abd72df5fa6fdd92b3d434c85ef529c1 WatchSource:0}: Error finding container fe4c075c800bd00af123a6041d39e194abd72df5fa6fdd92b3d434c85ef529c1: Status 404 returned error can't find the container with id fe4c075c800bd00af123a6041d39e194abd72df5fa6fdd92b3d434c85ef529c1
Nov 01 22:35:09.977702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:09.977676    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"]
Nov 01 22:35:09.982922 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:09.982894    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbfb0e7f_25ba_489f_aac2_258bd6e1966d.slice/crio-226f652c009a68e0d0da7b6c6927a52780aaa01c2cc0a7c6edc97bf5334a22b7 WatchSource:0}: Error finding container 226f652c009a68e0d0da7b6c6927a52780aaa01c2cc0a7c6edc97bf5334a22b7: Status 404 returned error can't find the container with id 226f652c009a68e0d0da7b6c6927a52780aaa01c2cc0a7c6edc97bf5334a22b7
Nov 01 22:35:10.253475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.253389    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" event={"ID":"2e2f1c24-981c-4224-834a-1169ee831c91","Type":"ContainerStarted","Data":"de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e"}
Nov 01 22:35:10.253475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.253445    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" event={"ID":"2e2f1c24-981c-4224-834a-1169ee831c91","Type":"ContainerStarted","Data":"fe4c075c800bd00af123a6041d39e194abd72df5fa6fdd92b3d434c85ef529c1"}
Nov 01 22:35:10.253686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.253476    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:10.254951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.254920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" event={"ID":"dbfb0e7f-25ba-489f-aac2-258bd6e1966d","Type":"ContainerStarted","Data":"18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9"}
Nov 01 22:35:10.255077 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.254967    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" event={"ID":"dbfb0e7f-25ba-489f-aac2-258bd6e1966d","Type":"ContainerStarted","Data":"226f652c009a68e0d0da7b6c6927a52780aaa01c2cc0a7c6edc97bf5334a22b7"}
Nov 01 22:35:10.255077 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.255020    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:10.363705 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.363647    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" podStartSLOduration=1.363629608 podStartE2EDuration="1.363629608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:10.361815513 +0000 UTC m=+691.679465513" watchObservedRunningTime="2026-11-01 22:35:10.363629608 +0000 UTC m=+691.681279610"
Nov 01 22:35:10.473090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:10.473033    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" podStartSLOduration=2.473012856 podStartE2EDuration="2.473012856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:10.468814279 +0000 UTC m=+691.786464291" watchObservedRunningTime="2026-11-01 22:35:10.473012856 +0000 UTC m=+691.790662857"
Nov 01 22:35:11.231924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.231894    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:11.428321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.428289    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:11.614718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.614627    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"]
Nov 01 22:35:11.618567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.618547    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.668833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.668805    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-e71d55\""
Nov 01 22:35:11.668833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.668806    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-e71d55-dockercfg-nvpdr\""
Nov 01 22:35:11.712803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.705641    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.713013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.712936    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g8gm\" (UniqueName: \"kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm\") pod \"payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.720742 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.720691    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"]
Nov 01 22:35:11.814594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.814557    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g8gm\" (UniqueName: \"kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm\") pod \"payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.814778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.814693    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.815558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.815530    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.872670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.872606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g8gm\" (UniqueName: \"kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm\") pod \"payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:11.932572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:11.932537    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:12.322384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:12.322355    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"]
Nov 01 22:35:12.326799 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:12.326769    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e351981_372f_4a68_ad4a_5cb2fe8ae610.slice/crio-6535231b7db4a88c0a9dcce5f9c43c2e23c145a87f0b187a4583ab6cc1fd057e WatchSource:0}: Error finding container 6535231b7db4a88c0a9dcce5f9c43c2e23c145a87f0b187a4583ab6cc1fd057e: Status 404 returned error can't find the container with id 6535231b7db4a88c0a9dcce5f9c43c2e23c145a87f0b187a4583ab6cc1fd057e
Nov 01 22:35:12.527054 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:35:12.527015    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:35:13.305355 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:13.305313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" event={"ID":"9e351981-372f-4a68-ad4a-5cb2fe8ae610","Type":"ContainerStarted","Data":"096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1"}
Nov 01 22:35:13.305536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:13.305375    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" event={"ID":"9e351981-372f-4a68-ad4a-5cb2fe8ae610","Type":"ContainerStarted","Data":"6535231b7db4a88c0a9dcce5f9c43c2e23c145a87f0b187a4583ab6cc1fd057e"}
Nov 01 22:35:13.305614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:13.305585    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:13.415619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:13.415565    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" podStartSLOduration=2.415552046 podStartE2EDuration="2.415552046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:13.414670618 +0000 UTC m=+694.732320620" watchObservedRunningTime="2026-11-01 22:35:13.415552046 +0000 UTC m=+694.733202047"
Nov 01 22:35:13.922377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:13.922341    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:14.149408 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:14.149371    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:15.112386 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:15.112336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:15.318876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:15.318845    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:17.238830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:17.238801    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 22:35:17.239320 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:17.238863    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 22:35:20.961750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:20.954999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:21.161123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.161087    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:21.265559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.265480    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 22:35:21.266380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.266329    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 22:35:21.352982 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.352915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:21.576402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.576323    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:21.778082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.778010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:21.803599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.803565    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"]
Nov 01 22:35:21.807261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.807244    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:21.858617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.858557    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-a6dd2f\""
Nov 01 22:35:21.907983 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.907903    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-a6dd2f-dockercfg-bpjhq\""
Nov 01 22:35:21.909124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.909098    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"]
Nov 01 22:35:21.917529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.917492    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:21.917955 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:21.917751    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lknbd\" (UniqueName: \"kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd\") pod \"payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:22.018558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:22.018517    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lknbd\" (UniqueName: \"kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd\") pod \"payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:22.019003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:22.018638    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:22.019575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:22.019552    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:22.072066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:22.072042    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lknbd\" (UniqueName: \"kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd\") pod \"payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:22.120576 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:22.120508    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:22.502892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:22.502858    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"]
Nov 01 22:35:22.507541 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:22.507509    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb902e77f_91d9_4f19_bfeb_f2fa1034b66f.slice/crio-f4b9c7b4dc22fc771c7028ec0149d72fcdcbbdc5a9ca7bab9e5ea7991800d3e3 WatchSource:0}: Error finding container f4b9c7b4dc22fc771c7028ec0149d72fcdcbbdc5a9ca7bab9e5ea7991800d3e3: Status 404 returned error can't find the container with id f4b9c7b4dc22fc771c7028ec0149d72fcdcbbdc5a9ca7bab9e5ea7991800d3e3
Nov 01 22:35:23.367801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:23.367763    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" event={"ID":"b902e77f-91d9-4f19-bfeb-f2fa1034b66f","Type":"ContainerStarted","Data":"fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c"}
Nov 01 22:35:23.368191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:23.367830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" event={"ID":"b902e77f-91d9-4f19-bfeb-f2fa1034b66f","Type":"ContainerStarted","Data":"f4b9c7b4dc22fc771c7028ec0149d72fcdcbbdc5a9ca7bab9e5ea7991800d3e3"}
Nov 01 22:35:23.368191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:23.367870    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:23.480277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:23.480229    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" podStartSLOduration=2.48021227 podStartE2EDuration="2.48021227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:23.48016575 +0000 UTC m=+704.797815751" watchObservedRunningTime="2026-11-01 22:35:23.48021227 +0000 UTC m=+704.797862264"
Nov 01 22:35:24.321607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:24.321579    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 22:35:26.362993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.362955    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:26.559945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.559913    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:26.760256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.760228    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:26.786691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.786657    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"]
Nov 01 22:35:26.791561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.791542    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:26.841950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.841930    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-c768f3\""
Nov 01 22:35:26.865832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.865790    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:26.865972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.865912    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q78q\" (UniqueName: \"kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q\") pod \"payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:26.887912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.887882    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-c768f3-dockercfg-86wpg\""
Nov 01 22:35:26.893967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.893933    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"]
Nov 01 22:35:26.946101 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.946071    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"]
Nov 01 22:35:26.949990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.949973    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:26.966508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.966486    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q78q\" (UniqueName: \"kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q\") pod \"payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:26.966623 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.966599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:26.967422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:26.967401    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:27.005944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.005914    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"]
Nov 01 22:35:27.021332 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.021280    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q78q\" (UniqueName: \"kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q\") pod \"payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:27.068121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.068099    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.068252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.068152    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwvt4\" (UniqueName: \"kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4\") pod \"payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.105143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.105118    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:27.169575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.169539    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.169716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.169648    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwvt4\" (UniqueName: \"kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4\") pod \"payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.171070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.171016    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.225098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.225067    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwvt4\" (UniqueName: \"kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4\") pod \"payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.263216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.263186    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:27.474115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.474085    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"]
Nov 01 22:35:27.478668 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:27.478645    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf789004b_4add_4e83_bc23_f81ff62a7b25.slice/crio-7958a66ebded33a1a96eeb380c6d6c294cd1c6d5af31dcd42cae324a9067d658 WatchSource:0}: Error finding container 7958a66ebded33a1a96eeb380c6d6c294cd1c6d5af31dcd42cae324a9067d658: Status 404 returned error can't find the container with id 7958a66ebded33a1a96eeb380c6d6c294cd1c6d5af31dcd42cae324a9067d658
Nov 01 22:35:27.635499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:27.635473    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"]
Nov 01 22:35:27.641302 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:27.641277    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f17394_841a_4aa3_ae19_88c12f1aaa2f.slice/crio-1381be3e455a8796900408e10853f06652fd5d23e4b1dd924e13e6f6af449b94 WatchSource:0}: Error finding container 1381be3e455a8796900408e10853f06652fd5d23e4b1dd924e13e6f6af449b94: Status 404 returned error can't find the container with id 1381be3e455a8796900408e10853f06652fd5d23e4b1dd924e13e6f6af449b94
Nov 01 22:35:28.400254 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.400218    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" event={"ID":"f789004b-4add-4e83-bc23-f81ff62a7b25","Type":"ContainerStarted","Data":"c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103"}
Nov 01 22:35:28.400476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.400277    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" event={"ID":"f789004b-4add-4e83-bc23-f81ff62a7b25","Type":"ContainerStarted","Data":"7958a66ebded33a1a96eeb380c6d6c294cd1c6d5af31dcd42cae324a9067d658"}
Nov 01 22:35:28.400476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.400307    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:28.401918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.401892    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" event={"ID":"a9f17394-841a-4aa3-ae19-88c12f1aaa2f","Type":"ContainerStarted","Data":"2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547"}
Nov 01 22:35:28.402034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.401934    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" event={"ID":"a9f17394-841a-4aa3-ae19-88c12f1aaa2f","Type":"ContainerStarted","Data":"1381be3e455a8796900408e10853f06652fd5d23e4b1dd924e13e6f6af449b94"}
Nov 01 22:35:28.402034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.401967    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:28.645958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.645908    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" podStartSLOduration=2.645892822 podStartE2EDuration="2.645892822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:28.53219254 +0000 UTC m=+709.849842540" watchObservedRunningTime="2026-11-01 22:35:28.645892822 +0000 UTC m=+709.963542872"
Nov 01 22:35:28.647431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:28.647394    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" podStartSLOduration=2.647384766 podStartE2EDuration="2.647384766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:28.646958778 +0000 UTC m=+709.964608762" watchObservedRunningTime="2026-11-01 22:35:28.647384766 +0000 UTC m=+709.965034815"
Nov 01 22:35:31.279581 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.279547    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:31.474994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.474956    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:31.662927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.662841    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:31.742323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.742293    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"]
Nov 01 22:35:31.746503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.746486    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:31.802128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.802101    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"]
Nov 01 22:35:31.824953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.824917    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:31.825234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.825211    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h66k\" (UniqueName: \"kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k\") pod \"payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:31.856751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.854977    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-9dfe9f\""
Nov 01 22:35:31.856751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.855603    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-9dfe9f-dockercfg-lqg8s\""
Nov 01 22:35:31.895889 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.895856    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"]
Nov 01 22:35:31.899953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.899932    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:31.926589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.926518    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:31.926589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.926570    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h66k\" (UniqueName: \"kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k\") pod \"payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:31.927248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.927223    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:31.953607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.953578    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"]
Nov 01 22:35:31.980619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:31.980597    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h66k\" (UniqueName: \"kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k\") pod \"payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:32.027793 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.027759    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpq9m\" (UniqueName: \"kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m\") pod \"payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.027910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.027831    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.060837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.060813    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:32.101870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.101835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:32.129491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.129460    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpq9m\" (UniqueName: \"kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m\") pod \"payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.129631 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.129541    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.130433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.130406    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.183823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.183770    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpq9m\" (UniqueName: \"kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m\") pod \"payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.214294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.214266    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:32.437631 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.437562    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"]
Nov 01 22:35:32.441311 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:32.441289    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fcb23f7_58a4_4c2b_bd70_d92a09214d42.slice/crio-0719a8f4d5ebe1cdf9bca4ac87d5e689513a1ba36b7f5bccdadcfa78b3ce6ae2 WatchSource:0}: Error finding container 0719a8f4d5ebe1cdf9bca4ac87d5e689513a1ba36b7f5bccdadcfa78b3ce6ae2: Status 404 returned error can't find the container with id 0719a8f4d5ebe1cdf9bca4ac87d5e689513a1ba36b7f5bccdadcfa78b3ce6ae2
Nov 01 22:35:32.587270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.587237    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"]
Nov 01 22:35:32.591834 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:32.591805    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a8102a_eda2_4ae1_ab5d_a59b4555efcb.slice/crio-4e1e786b8196ee7fb171731eb6b9cc11bc8f17505b67e21e8646b43fd919e1a1 WatchSource:0}: Error finding container 4e1e786b8196ee7fb171731eb6b9cc11bc8f17505b67e21e8646b43fd919e1a1: Status 404 returned error can't find the container with id 4e1e786b8196ee7fb171731eb6b9cc11bc8f17505b67e21e8646b43fd919e1a1
Nov 01 22:35:32.683129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.683098    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:32.897346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:32.897310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:33.433510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.433474    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" event={"ID":"79a8102a-eda2-4ae1-ab5d-a59b4555efcb","Type":"ContainerStarted","Data":"53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f"}
Nov 01 22:35:33.433719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.433537    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" event={"ID":"79a8102a-eda2-4ae1-ab5d-a59b4555efcb","Type":"ContainerStarted","Data":"4e1e786b8196ee7fb171731eb6b9cc11bc8f17505b67e21e8646b43fd919e1a1"}
Nov 01 22:35:33.433719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.433584    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:33.435070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.435041    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" event={"ID":"9fcb23f7-58a4-4c2b-bd70-d92a09214d42","Type":"ContainerStarted","Data":"338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee"}
Nov 01 22:35:33.435191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.435089    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" event={"ID":"9fcb23f7-58a4-4c2b-bd70-d92a09214d42","Type":"ContainerStarted","Data":"0719a8f4d5ebe1cdf9bca4ac87d5e689513a1ba36b7f5bccdadcfa78b3ce6ae2"}
Nov 01 22:35:33.435191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.435130    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:33.548131 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.548089    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" podStartSLOduration=2.548077143 podStartE2EDuration="2.548077143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:33.545335494 +0000 UTC m=+714.862985507" watchObservedRunningTime="2026-11-01 22:35:33.548077143 +0000 UTC m=+714.865727145"
Nov 01 22:35:33.653878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:33.653830    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" podStartSLOduration=2.653818708 podStartE2EDuration="2.653818708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:33.652852348 +0000 UTC m=+714.970502349" watchObservedRunningTime="2026-11-01 22:35:33.653818708 +0000 UTC m=+714.971468733"
Nov 01 22:35:34.383406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:34.382404    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 22:35:36.301473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:36.301426    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:36.679529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:36.679452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:36.896290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:36.896259    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:37.176190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:37.176157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:39.412493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:39.412454    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 22:35:39.413904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:39.413879    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 22:35:41.867494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.867463    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"]
Nov 01 22:35:41.876521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.876498    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:41.927830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.927804    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"]
Nov 01 22:35:41.936138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.936106    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:41.936410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.936374    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84hwf\" (UniqueName: \"kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:41.936530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.936505    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:41.936588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.936556    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:41.969875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.969827    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c54de0-serving-cert\""
Nov 01 22:35:41.970015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.969885    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-xp822\""
Nov 01 22:35:41.970015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:41.969989    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 22:35:42.038434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.038404    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84hwf\" (UniqueName: \"kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.038676 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.038656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.038833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.038818    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.039011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.038994    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.039897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.039873    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.040419 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:35:42.040399    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-serving-cert: secret "maas-api-e2e-worker-w0-c54de0-serving-cert" not found
Nov 01 22:35:42.040581 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:35:42.040570    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls podName:ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5 nodeName:}" failed. No retries permitted until 2026-11-01 22:35:42.540543001 +0000 UTC m=+723.858192994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls") pod "maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" (UID: "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5") : secret "maas-api-e2e-worker-w0-c54de0-serving-cert" not found
Nov 01 22:35:42.043862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.043803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.102030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.101991    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84hwf\" (UniqueName: \"kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.122471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.122402    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:42.313433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.313395    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:42.503067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.503029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:42.526219 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:35:42.526188    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:35:42.543354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.543326    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.545832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.545807    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:42.793052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:42.792979    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:43.176659 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:43.176585    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"]
Nov 01 22:35:43.180325 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:35:43.180296    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd4cda4_c874_4922_a2fc_a39b4cfc2fe5.slice/crio-13682ddd613b7952170f0cf8f05441e514ecea2f6ab624e0bd5e890aff507a20 WatchSource:0}: Error finding container 13682ddd613b7952170f0cf8f05441e514ecea2f6ab624e0bd5e890aff507a20: Status 404 returned error can't find the container with id 13682ddd613b7952170f0cf8f05441e514ecea2f6ab624e0bd5e890aff507a20
Nov 01 22:35:43.505813 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:43.505785    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" event={"ID":"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5","Type":"ContainerStarted","Data":"13682ddd613b7952170f0cf8f05441e514ecea2f6ab624e0bd5e890aff507a20"}
Nov 01 22:35:44.444395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:44.444317    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 22:35:44.444895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:44.444479    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 22:35:44.514567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:44.514537    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" event={"ID":"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5","Type":"ContainerStarted","Data":"56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70"}
Nov 01 22:35:44.514668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:44.514592    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:44.778506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:44.778417    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" podStartSLOduration=2.962037681 podStartE2EDuration="3.778397115s" totalImagesPullingTime="816.359434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:35:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:35:44.776456827 +0000 UTC m=+726.094106823" watchObservedRunningTime="2026-11-01 22:35:44.778397115 +0000 UTC m=+726.096047108"
Nov 01 22:35:47.465508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:47.465473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:47.649042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:47.649008    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:50.527585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:50.527558    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 22:35:52.528457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:52.528390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:35:57.353412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:35:57.353382    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:02.252809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:02.252770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:02.841980 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:02.841911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:04.201398 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:04.201356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:04.847536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:04.847385    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:05.076355 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:05.076301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:07.562531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:07.562501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:09.017118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:09.017083    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:09.567161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:09.567128    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:09.761965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:09.761925    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:12.526113 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:36:12.526074    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:36:12.944484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:12.944401    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:18.205504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:18.205468    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:20.780103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:20.780022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:21.324035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:21.323992    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:21.518975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:21.518929    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:23.622543 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:23.622507    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:25.963461 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:25.963422    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:26.172266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:26.172226    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:26.493129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:26.493099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:26.697957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:26.697923    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:28.959114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:28.959075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:30.057302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:30.057267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:30.775913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:30.775874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:31.005515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:31.005479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:31.730515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:31.730473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:31.924447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:31.924411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:34.415332 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:34.415295    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:35.824806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:35.824770    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:36.460843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:36.460808    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:36.665323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:36.665292    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:38.944104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:38.944077    4531 scope.go:122] "RemoveContainer" containerID="95b817555027dd3ee6f4e2beee1f6bc9d3165a72a0010104cd5d26eafb2e13c9"
Nov 01 22:36:39.747516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:39.747480    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:42.295680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.295644    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"]
Nov 01 22:36:42.300967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.300947    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.350519 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.350494    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"default-dockercfg-l6sl9\""
Nov 01 22:36:42.350709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.350520    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"openshift-service-ca.crt\""
Nov 01 22:36:42.350811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.350537    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"kube-root-ca.crt\""
Nov 01 22:36:42.396819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.396790    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"simulator-w2-c768f3-kserve-self-signed-certs\""
Nov 01 22:36:42.400272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.400242    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"]
Nov 01 22:36:42.422294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.422268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.422410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.422308    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.422462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.422398    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.422462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.422454    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.422586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.422561    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6d65\" (UniqueName: \"kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.422641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.422600    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.432115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.432086    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:42.524199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524162    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.524399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6d65\" (UniqueName: \"kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.524399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524250    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.524399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524359    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.524399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524387    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.524640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524427    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.524908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524887    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.525066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.524921    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.525301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.525280    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.526525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.526501    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.527228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.527205    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.527635 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:36:42.527606    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:36:42.578401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.578325    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6d65\" (UniqueName: \"kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65\") pod \"simulator-w2-c768f3-kserve-6966f965f6-bcpf5\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") " pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.613413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.613373    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:42.998736 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:42.998703    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"]
Nov 01 22:36:43.003815 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:36:43.003792    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod39ebfba5_06df_4469_8a21_8fd8f562dcea.slice/crio-1b3436e54e30de42b65b968ccd180920340db58ace6170553b20f57e96fbf677 WatchSource:0}: Error finding container 1b3436e54e30de42b65b968ccd180920340db58ace6170553b20f57e96fbf677: Status 404 returned error can't find the container with id 1b3436e54e30de42b65b968ccd180920340db58ace6170553b20f57e96fbf677
Nov 01 22:36:43.925095 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:43.925026    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" event={"ID":"39ebfba5-06df-4469-8a21-8fd8f562dcea","Type":"ContainerStarted","Data":"1b3436e54e30de42b65b968ccd180920340db58ace6170553b20f57e96fbf677"}
Nov 01 22:36:44.552860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:44.550419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:45.941962 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:45.941922    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" event={"ID":"39ebfba5-06df-4469-8a21-8fd8f562dcea","Type":"ContainerStarted","Data":"93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d"}
Nov 01 22:36:46.054699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:46.054648    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podStartSLOduration=2.054873742 podStartE2EDuration="4.054635715s" totalImagesPullingTime="1.999761973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:36:46.053018309 +0000 UTC m=+787.370668310" watchObservedRunningTime="2026-11-01 22:36:46.054635715 +0000 UTC m=+787.372285715"
Nov 01 22:36:46.614362 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:46.614332    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:49.366891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:49.366856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:52.613739 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:52.613702    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:52.626654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:52.626628    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:53.002005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:53.001979    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:36:54.228799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:54.228704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:36:59.053448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:36:59.053411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:00.243808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.243772    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"]
Nov 01 22:37:00.247541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.247525    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.311753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.311708    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"]
Nov 01 22:37:00.353031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.352972    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"default-dockercfg-wcfqv\""
Nov 01 22:37:00.353172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.353118    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"kube-root-ca.crt\""
Nov 01 22:37:00.353627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.353605    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"simulator-w4-9dfe9f-kserve-self-signed-certs\""
Nov 01 22:37:00.353779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.353628    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"openshift-service-ca.crt\""
Nov 01 22:37:00.410421 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.410386    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.410578 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.410427    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cw2\" (UniqueName: \"kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.410623 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.410565    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.410680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.410616    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.410811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.410794    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.410885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.410862    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.420600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.420571    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:00.512249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512163    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512374    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512441    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cw2\" (UniqueName: \"kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512503    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512536    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512598    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.512959 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.512935    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.513335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.513308    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.514821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.514799    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.515097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.515074    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.575532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.575503    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cw2\" (UniqueName: \"kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2\") pod \"simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") " pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:00.859352 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:00.859274    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:01.232121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:01.232086    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"]
Nov 01 22:37:01.236350 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:37:01.236320    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaafc9355_289e_42be_ba80_2514305c26de.slice/crio-892562fd7ec569a019f1f4865fb462f105764f0b6514bcd0bfdf83f07b915936 WatchSource:0}: Error finding container 892562fd7ec569a019f1f4865fb462f105764f0b6514bcd0bfdf83f07b915936: Status 404 returned error can't find the container with id 892562fd7ec569a019f1f4865fb462f105764f0b6514bcd0bfdf83f07b915936
Nov 01 22:37:02.056567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:02.056532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" event={"ID":"aafc9355-289e-42be-ba80-2514305c26de","Type":"ContainerStarted","Data":"ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91"}
Nov 01 22:37:02.056974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:02.056579    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" event={"ID":"aafc9355-289e-42be-ba80-2514305c26de","Type":"ContainerStarted","Data":"892562fd7ec569a019f1f4865fb462f105764f0b6514bcd0bfdf83f07b915936"}
Nov 01 22:37:02.164617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:02.164568    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podStartSLOduration=2.164555633 podStartE2EDuration="2.164555633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:02.16321614 +0000 UTC m=+803.480866143" watchObservedRunningTime="2026-11-01 22:37:02.164555633 +0000 UTC m=+803.482205635"
Nov 01 22:37:02.859523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:02.859494    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:04.040561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:04.040529    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:05.673315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.673279    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"]
Nov 01 22:37:05.677690 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.677669    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.766796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.766771    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.766975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.766931    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.767052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.767031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd7wb\" (UniqueName: \"kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.774525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.774503    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:37:05.774525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.774504    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:37:05.779640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.779617    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"]
Nov 01 22:37:05.867833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.867803    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.867973 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.867876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd7wb\" (UniqueName: \"kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.867973 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.867954    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.868496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.868475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.870321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.870295    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.921168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.921140    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd7wb\" (UniqueName: \"kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb\") pod \"maas-api-revoke-sub-3e0d63fe-4r2v7\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:05.990887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:05.990862    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:06.362993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:06.362211    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"]
Nov 01 22:37:06.367015 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:37:06.366986    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0201c0d_aac6_4d65_b628_8d43107ce507.slice/crio-8d2c97a23ab1de76dcbe3b5367bc8a0d002b6033ab90b3d6a89911cf4b9dbcba WatchSource:0}: Error finding container 8d2c97a23ab1de76dcbe3b5367bc8a0d002b6033ab90b3d6a89911cf4b9dbcba: Status 404 returned error can't find the container with id 8d2c97a23ab1de76dcbe3b5367bc8a0d002b6033ab90b3d6a89911cf4b9dbcba
Nov 01 22:37:07.095893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:07.095859    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7" event={"ID":"e0201c0d-aac6-4d65-b628-8d43107ce507","Type":"ContainerStarted","Data":"8d2c97a23ab1de76dcbe3b5367bc8a0d002b6033ab90b3d6a89911cf4b9dbcba"}
Nov 01 22:37:08.848769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:08.848715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:09.103054 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:09.102911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:10.859797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:10.859764    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:10.873756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:10.873739    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:11.137709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:11.137675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7" event={"ID":"e0201c0d-aac6-4d65-b628-8d43107ce507","Type":"ContainerStarted","Data":"3dcad0493aafefa2c9193d983f951e768e5d7eaec094f54f39f790a579fc6bbf"}
Nov 01 22:37:11.149606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:11.149580    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:37:11.247488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:11.247444    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7" podStartSLOduration=1.860297774 podStartE2EDuration="6.247431721s" totalImagesPullingTime="4.387133947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:11.245037767 +0000 UTC m=+812.562687768" watchObservedRunningTime="2026-11-01 22:37:11.247431721 +0000 UTC m=+812.565081722"
Nov 01 22:37:12.146633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:12.146591    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0201c0d-aac6-4d65-b628-8d43107ce507" containerID="3dcad0493aafefa2c9193d983f951e768e5d7eaec094f54f39f790a579fc6bbf" exitCode=0
Nov 01 22:37:12.147091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:12.146631    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7" event={"ID":"e0201c0d-aac6-4d65-b628-8d43107ce507","Type":"ContainerDied","Data":"3dcad0493aafefa2c9193d983f951e768e5d7eaec094f54f39f790a579fc6bbf"}
Nov 01 22:37:12.525066 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:37:12.525034    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:37:12.863572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:12.863493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:13.326429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.326406    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:13.455174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.455146    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca\") pod \"e0201c0d-aac6-4d65-b628-8d43107ce507\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") "
Nov 01 22:37:13.455347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.455198    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb\" (UniqueName: \"kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb\") pod \"e0201c0d-aac6-4d65-b628-8d43107ce507\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") "
Nov 01 22:37:13.455347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.455337    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca\") pod \"e0201c0d-aac6-4d65-b628-8d43107ce507\" (UID: \"e0201c0d-aac6-4d65-b628-8d43107ce507\") "
Nov 01 22:37:13.455489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.455466    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca" pod "e0201c0d-aac6-4d65-b628-8d43107ce507" (UID: "e0201c0d-aac6-4d65-b628-8d43107ce507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:37:13.455649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.455624    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:13.457382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.457357    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca" pod "e0201c0d-aac6-4d65-b628-8d43107ce507" (UID: "e0201c0d-aac6-4d65-b628-8d43107ce507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:37:13.457382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.457361    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb" pod "e0201c0d-aac6-4d65-b628-8d43107ce507" (UID: "e0201c0d-aac6-4d65-b628-8d43107ce507"). InnerVolumeSpecName "kube-api-access-pd7wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:37:13.556605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.556576    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd7wb\" (UniqueName: \"kubernetes.io/projected/e0201c0d-aac6-4d65-b628-8d43107ce507-kube-api-access-pd7wb\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:13.556605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.556603    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0201c0d-aac6-4d65-b628-8d43107ce507-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:13.727651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:13.727566    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:14.164577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:14.164504    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"
Nov 01 22:37:14.164577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:14.164526    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7" event={"ID":"e0201c0d-aac6-4d65-b628-8d43107ce507","Type":"ContainerDied","Data":"8d2c97a23ab1de76dcbe3b5367bc8a0d002b6033ab90b3d6a89911cf4b9dbcba"}
Nov 01 22:37:14.164577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:14.164567    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2c97a23ab1de76dcbe3b5367bc8a0d002b6033ab90b3d6a89911cf4b9dbcba"
Nov 01 22:37:18.577615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:18.577577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:23.441930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:23.441899    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:28.252041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:28.251997    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:29.928696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:29.928649    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:32.082737 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:32.082693    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:33.143602 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:33.143563    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:35.922515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:35.922480    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"]
Nov 01 22:37:35.923122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:35.923103    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e0201c0d-aac6-4d65-b628-8d43107ce507" containerName="revoke-keys"
Nov 01 22:37:35.923173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:35.923123    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0201c0d-aac6-4d65-b628-8d43107ce507" containerName="revoke-keys"
Nov 01 22:37:35.923253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:35.923243    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0201c0d-aac6-4d65-b628-8d43107ce507" containerName="revoke-keys"
Nov 01 22:37:35.927146 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:35.927130    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:35.982674 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:35.982644    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"]
Nov 01 22:37:36.024575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.024549    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:37:36.024830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.024814    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:37:36.091526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.091494    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.091656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.091610    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.091751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.091675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.192570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.192497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.192570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.192554    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.192817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.192597    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.193364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.193335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.195060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.195033    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.246293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.246266    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz\") pod \"maas-api-revoke-sub-073ddf60-b6zn4\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.540492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.540465    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:36.913935 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:36.913849    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"]
Nov 01 22:37:36.917900 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:37:36.917872    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2764ac6_d90e_41bf_b68d_936925bf8d55.slice/crio-3c3003e8e7b3766979ba7537c92b9eb85fba189e2cf95182183ff3c104ea1aad WatchSource:0}: Error finding container 3c3003e8e7b3766979ba7537c92b9eb85fba189e2cf95182183ff3c104ea1aad: Status 404 returned error can't find the container with id 3c3003e8e7b3766979ba7537c92b9eb85fba189e2cf95182183ff3c104ea1aad
Nov 01 22:37:37.325769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:37.325705    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4" event={"ID":"b2764ac6-d90e-41bf-b68d-936925bf8d55","Type":"ContainerStarted","Data":"2668545ba20499ee823da63ae5ca3d05be909314db1b4acc0bb742d5da49e40c"}
Nov 01 22:37:37.326124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:37.325783    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4" event={"ID":"b2764ac6-d90e-41bf-b68d-936925bf8d55","Type":"ContainerStarted","Data":"3c3003e8e7b3766979ba7537c92b9eb85fba189e2cf95182183ff3c104ea1aad"}
Nov 01 22:37:37.438261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:37.438169    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4" podStartSLOduration=2.438156218 podStartE2EDuration="2.438156218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:37.434518547 +0000 UTC m=+838.752168574" watchObservedRunningTime="2026-11-01 22:37:37.438156218 +0000 UTC m=+838.755806218"
Nov 01 22:37:38.071172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:38.071087    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:38.334244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:38.334160    4531 generic.go:356] "Generic (PLEG): container finished" podID="b2764ac6-d90e-41bf-b68d-936925bf8d55" containerID="2668545ba20499ee823da63ae5ca3d05be909314db1b4acc0bb742d5da49e40c" exitCode=0
Nov 01 22:37:38.334244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:38.334207    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4" event={"ID":"b2764ac6-d90e-41bf-b68d-936925bf8d55","Type":"ContainerDied","Data":"2668545ba20499ee823da63ae5ca3d05be909314db1b4acc0bb742d5da49e40c"}
Nov 01 22:37:39.483171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.483145    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:39.627679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.627613    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca\") pod \"b2764ac6-d90e-41bf-b68d-936925bf8d55\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") "
Nov 01 22:37:39.627679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.627652    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca\") pod \"b2764ac6-d90e-41bf-b68d-936925bf8d55\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") "
Nov 01 22:37:39.627864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.627696    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz\") pod \"b2764ac6-d90e-41bf-b68d-936925bf8d55\" (UID: \"b2764ac6-d90e-41bf-b68d-936925bf8d55\") "
Nov 01 22:37:39.628032 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.628005    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca" pod "b2764ac6-d90e-41bf-b68d-936925bf8d55" (UID: "b2764ac6-d90e-41bf-b68d-936925bf8d55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:37:39.629653 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.629630    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca" pod "b2764ac6-d90e-41bf-b68d-936925bf8d55" (UID: "b2764ac6-d90e-41bf-b68d-936925bf8d55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:37:39.629776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.629750    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz" pod "b2764ac6-d90e-41bf-b68d-936925bf8d55" (UID: "b2764ac6-d90e-41bf-b68d-936925bf8d55"). InnerVolumeSpecName "kube-api-access-jtwjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:37:39.729097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.729070    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtwjz\" (UniqueName: \"kubernetes.io/projected/b2764ac6-d90e-41bf-b68d-936925bf8d55-kube-api-access-jtwjz\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:39.729097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.729096    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:39.729289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:39.729107    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2764ac6-d90e-41bf-b68d-936925bf8d55-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:40.349190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:40.349158    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4" event={"ID":"b2764ac6-d90e-41bf-b68d-936925bf8d55","Type":"ContainerDied","Data":"3c3003e8e7b3766979ba7537c92b9eb85fba189e2cf95182183ff3c104ea1aad"}
Nov 01 22:37:40.349391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:40.349205    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c3003e8e7b3766979ba7537c92b9eb85fba189e2cf95182183ff3c104ea1aad"
Nov 01 22:37:40.349391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:40.349169    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"
Nov 01 22:37:40.723809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:40.723781    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:40.926045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:40.926012    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:42.525198 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:37:42.525164    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:37:42.847456 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:42.847339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:47.659317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:47.659286    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:51.209756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.209656    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"]
Nov 01 22:37:51.210452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.210430    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b2764ac6-d90e-41bf-b68d-936925bf8d55" containerName="revoke-keys"
Nov 01 22:37:51.210452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.210452    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2764ac6-d90e-41bf-b68d-936925bf8d55" containerName="revoke-keys"
Nov 01 22:37:51.210626 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.210614    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2764ac6-d90e-41bf-b68d-936925bf8d55" containerName="revoke-keys"
Nov 01 22:37:51.218189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.218170    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.236555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.236529    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ftt9\" (UniqueName: \"kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.236663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.236600    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.236762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.236746    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.270719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.270690    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"]
Nov 01 22:37:51.311896 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.311872    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:37:51.312016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.312002    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:37:51.337971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.337945    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.338108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.337984    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ftt9\" (UniqueName: \"kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.338108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.338026    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.338844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.338816    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.340514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.340490    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.392232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.392200    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ftt9\" (UniqueName: \"kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9\") pod \"maas-api-revoke-sub-2523a479-h56v5\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.553472 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.553356    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:51.935420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:51.935339    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"]
Nov 01 22:37:51.940190 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:37:51.940160    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8bf1ce1_16e2_4b28_89e1_6ae37bd9350a.slice/crio-9e4e4319ae145275c65c7c5054aea4986727c5500171622d46ea78412100500f WatchSource:0}: Error finding container 9e4e4319ae145275c65c7c5054aea4986727c5500171622d46ea78412100500f: Status 404 returned error can't find the container with id 9e4e4319ae145275c65c7c5054aea4986727c5500171622d46ea78412100500f
Nov 01 22:37:52.431998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:52.431956    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5" event={"ID":"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a","Type":"ContainerStarted","Data":"e01f5b5c887aa4680dcc2ac85fe348e32a83013ce5415fe0851ee65036a394ef"}
Nov 01 22:37:52.432459 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:52.432049    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5" event={"ID":"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a","Type":"ContainerStarted","Data":"9e4e4319ae145275c65c7c5054aea4986727c5500171622d46ea78412100500f"}
Nov 01 22:37:52.469497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:52.469457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:52.542531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:52.542468    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5" podStartSLOduration=1.5424499379999999 podStartE2EDuration="1.542449938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:52.539547773 +0000 UTC m=+853.857197775" watchObservedRunningTime="2026-11-01 22:37:52.542449938 +0000 UTC m=+853.860099940"
Nov 01 22:37:53.440934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.440854    4531 generic.go:356] "Generic (PLEG): container finished" podID="a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" containerID="e01f5b5c887aa4680dcc2ac85fe348e32a83013ce5415fe0851ee65036a394ef" exitCode=0
Nov 01 22:37:53.441307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.440938    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5" event={"ID":"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a","Type":"ContainerDied","Data":"e01f5b5c887aa4680dcc2ac85fe348e32a83013ce5415fe0851ee65036a394ef"}
Nov 01 22:37:53.670380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.670346    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"]
Nov 01 22:37:53.674476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.674456    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.728102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.728073    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"openshift-service-ca.crt\""
Nov 01 22:37:53.728279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.728156    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"default-dockercfg-8s5xm\""
Nov 01 22:37:53.729234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.729209    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"premium-w5-d6101b-kserve-self-signed-certs\""
Nov 01 22:37:53.733625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.733598    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"]
Nov 01 22:37:53.737430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.737409    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"kube-root-ca.crt\""
Nov 01 22:37:53.761349 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.761321    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.761457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.761362    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.761457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.761380    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kpsh\" (UniqueName: \"kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.761574 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.761553    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.761614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.761596    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.761753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.761718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.812758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.812699    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:37:53.863257 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863222    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863264    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kpsh\" (UniqueName: \"kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863338    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863373    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863454    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863524    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863819    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863859    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.863948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.863883    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.865993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.865969    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.866331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.866311    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.920658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.920627    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kpsh\" (UniqueName: \"kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh\") pod \"premium-w5-d6101b-kserve-54b97dd857-28875\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") " pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:53.986634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:53.986553    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:54.383871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.383802    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"]
Nov 01 22:37:54.387546 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:37:54.387524    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d9d7fb2_7701_49f7_9184_7d85caba1ed1.slice/crio-d41e78100fc7d3ba8de471c135923149058034c489fe886918bf39d13d35b4fa WatchSource:0}: Error finding container d41e78100fc7d3ba8de471c135923149058034c489fe886918bf39d13d35b4fa: Status 404 returned error can't find the container with id d41e78100fc7d3ba8de471c135923149058034c489fe886918bf39d13d35b4fa
Nov 01 22:37:54.449415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.449388    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" event={"ID":"7d9d7fb2-7701-49f7-9184-7d85caba1ed1","Type":"ContainerStarted","Data":"21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d"}
Nov 01 22:37:54.449811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.449430    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" event={"ID":"7d9d7fb2-7701-49f7-9184-7d85caba1ed1","Type":"ContainerStarted","Data":"d41e78100fc7d3ba8de471c135923149058034c489fe886918bf39d13d35b4fa"}
Nov 01 22:37:54.564039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.563991    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podStartSLOduration=1.563976631 podStartE2EDuration="1.563976631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:37:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:37:54.56003531 +0000 UTC m=+855.877685331" watchObservedRunningTime="2026-11-01 22:37:54.563976631 +0000 UTC m=+855.881626631"
Nov 01 22:37:54.575561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.575537    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:54.671176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.671113    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca\") pod \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") "
Nov 01 22:37:54.671288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.671266    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9\" (UniqueName: \"kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9\") pod \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") "
Nov 01 22:37:54.671329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.671303    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca\") pod \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\" (UID: \"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a\") "
Nov 01 22:37:54.671428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.671408    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca" pod "a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" (UID: "a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:37:54.671765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.671721    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:54.673306 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.673283    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca" pod "a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" (UID: "a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:37:54.673452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.673438    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9" pod "a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" (UID: "a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a"). InnerVolumeSpecName "kube-api-access-6ftt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:37:54.773002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.772964    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ftt9\" (UniqueName: \"kubernetes.io/projected/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-kube-api-access-6ftt9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:54.773165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.773024    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:37:54.987227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:54.987198    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:37:55.458097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:55.458067    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"
Nov 01 22:37:55.458478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:55.458091    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5" event={"ID":"a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a","Type":"ContainerDied","Data":"9e4e4319ae145275c65c7c5054aea4986727c5500171622d46ea78412100500f"}
Nov 01 22:37:55.458478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:55.458141    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4e4319ae145275c65c7c5054aea4986727c5500171622d46ea78412100500f"
Nov 01 22:37:57.425062 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:37:57.425005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:02.231087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:02.231043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:03.987254 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:03.987216    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:38:04.000699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:04.000670    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:04.008544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:04.008510    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:38:04.537064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:04.537035    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:38:06.773442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.773401    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"]
Nov 01 22:38:06.773928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.773917    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" containerName="revoke-keys"
Nov 01 22:38:06.773976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.773930    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" containerName="revoke-keys"
Nov 01 22:38:06.774053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.774042    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" containerName="revoke-keys"
Nov 01 22:38:06.777501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.777485    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.828930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.828905    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:38:06.833383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.833360    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"]
Nov 01 22:38:06.875158 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.875138    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:38:06.895192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.895164    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.895346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.895329    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4rpj\" (UniqueName: \"kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.895411 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.895378    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.996214 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.996182    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.996480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.996274    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.996480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.996346    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4rpj\" (UniqueName: \"kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.997038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.997014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:06.998719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:06.998691    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:07.006757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:07.006712    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:07.052100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:07.052039    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4rpj\" (UniqueName: \"kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj\") pod \"maas-api-revoke-sub-5e2a3bb4-cxtp4\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:07.091185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:07.091159    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:07.473469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:07.473438    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"]
Nov 01 22:38:07.477995 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:38:07.477966    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec944f0f_cc3a_4a19_b469_98a8dd10b0d2.slice/crio-83b233d082e78274a30a493eff84fcb4b0ff781509bcf04b1e0ad7cb28dc0905 WatchSource:0}: Error finding container 83b233d082e78274a30a493eff84fcb4b0ff781509bcf04b1e0ad7cb28dc0905: Status 404 returned error can't find the container with id 83b233d082e78274a30a493eff84fcb4b0ff781509bcf04b1e0ad7cb28dc0905
Nov 01 22:38:07.545893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:07.545870    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4" event={"ID":"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2","Type":"ContainerStarted","Data":"83b233d082e78274a30a493eff84fcb4b0ff781509bcf04b1e0ad7cb28dc0905"}
Nov 01 22:38:08.554442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:08.554413    4531 generic.go:356] "Generic (PLEG): container finished" podID="ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" containerID="0a5bef3d495d10d5dd59bf11ece6d4dcb3b532018692458a10b05f7b2fa2832f" exitCode=0
Nov 01 22:38:08.554827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:08.554453    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4" event={"ID":"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2","Type":"ContainerDied","Data":"0a5bef3d495d10d5dd59bf11ece6d4dcb3b532018692458a10b05f7b2fa2832f"}
Nov 01 22:38:09.716065 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.716043    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:09.823479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.823410    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca\") pod \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") "
Nov 01 22:38:09.823619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.823561    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca\") pod \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") "
Nov 01 22:38:09.823690 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.823673    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj\" (UniqueName: \"kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj\") pod \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\" (UID: \"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2\") "
Nov 01 22:38:09.823849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.823825    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca" pod "ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" (UID: "ec944f0f-cc3a-4a19-b469-98a8dd10b0d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:38:09.824063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.824038    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:09.826399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.826356    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj" pod "ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" (UID: "ec944f0f-cc3a-4a19-b469-98a8dd10b0d2"). InnerVolumeSpecName "kube-api-access-n4rpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:38:09.827360 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.827335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca" pod "ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" (UID: "ec944f0f-cc3a-4a19-b469-98a8dd10b0d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:38:09.925589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.925561    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:09.925707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:09.925594    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4rpj\" (UniqueName: \"kubernetes.io/projected/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2-kube-api-access-n4rpj\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:10.569310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:10.569277    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"
Nov 01 22:38:10.569553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:10.569323    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4" event={"ID":"ec944f0f-cc3a-4a19-b469-98a8dd10b0d2","Type":"ContainerDied","Data":"83b233d082e78274a30a493eff84fcb4b0ff781509bcf04b1e0ad7cb28dc0905"}
Nov 01 22:38:10.569553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:10.569370    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83b233d082e78274a30a493eff84fcb4b0ff781509bcf04b1e0ad7cb28dc0905"
Nov 01 22:38:11.872586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:11.872550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:12.525409 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:38:12.525373    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:38:16.529758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:16.529714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:19.710152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:19.710118    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:21.376644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:21.376608    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:24.729369 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.729337    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"]
Nov 01 22:38:24.729907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.729892    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" containerName="revoke-keys"
Nov 01 22:38:24.729958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.729908    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" containerName="revoke-keys"
Nov 01 22:38:24.729993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.729985    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" containerName="revoke-keys"
Nov 01 22:38:24.733410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.733395    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.782433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.782411    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5b8fd2-openshift-default-dockercfg-bqfks\""
Nov 01 22:38:24.843196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.842779    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"]
Nov 01 22:38:24.877543 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.877514    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2rcg\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.877820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.877571    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.877820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.877719    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.877820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.877794    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.878023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.877843    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.878023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.877963    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.878121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.878019    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.878121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.878067    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.878230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.878180    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.978874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.978840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.978874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.978878    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2rcg\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.978907    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.978987    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979046    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979089    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979142    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979194    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979381    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979382    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979763    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979822    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.979986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.979859    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.980373 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.980351    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.981481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.981457    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:24.982476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:24.982456    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:25.033293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.033269    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:25.033542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.033520    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2rcg\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:25.047178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.047146    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:25.482363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.482134    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"]
Nov 01 22:38:25.484651 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:38:25.484624    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ea652e_3239_40dd_9c46_c826105e70ed.slice/crio-ba2171cec845be78d27bef6f9b7ded5ad88209fb397b55aa005b6ce7aaae73c8 WatchSource:0}: Error finding container ba2171cec845be78d27bef6f9b7ded5ad88209fb397b55aa005b6ce7aaae73c8: Status 404 returned error can't find the container with id ba2171cec845be78d27bef6f9b7ded5ad88209fb397b55aa005b6ce7aaae73c8
Nov 01 22:38:25.487454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.487422    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:38:25.487561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.487501    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:38:25.487561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.487545    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:38:25.697340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.696972    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" event={"ID":"30ea652e-3239-40dd-9c46-c826105e70ed","Type":"ContainerStarted","Data":"a5f3a34e0d53fafe5ce608b8db3f5fa503da8c6a610d460fa7110f0139f37c59"}
Nov 01 22:38:25.700901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.700862    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" event={"ID":"30ea652e-3239-40dd-9c46-c826105e70ed","Type":"ContainerStarted","Data":"ba2171cec845be78d27bef6f9b7ded5ad88209fb397b55aa005b6ce7aaae73c8"}
Nov 01 22:38:25.832224 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:25.832117    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podStartSLOduration=1.832093606 podStartE2EDuration="1.832093606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:25.828412854 +0000 UTC m=+887.146062858" watchObservedRunningTime="2026-11-01 22:38:25.832093606 +0000 UTC m=+887.149743607"
Nov 01 22:38:26.048754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:26.048397    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:26.049315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:26.049284    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:38:26.049463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:26.049341    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 22:38:26.203495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:26.203443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:27.048405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.048371    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:38:27.048812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.048422    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 22:38:27.286752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.285802    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"]
Nov 01 22:38:27.291094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.291071    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.392144 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.392116    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:38:27.392357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.392342    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:38:27.395577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.395551    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"]
Nov 01 22:38:27.412251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.412215    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.412395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.412292    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.412532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.412512    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5kkd\" (UniqueName: \"kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.513545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.513511    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5kkd\" (UniqueName: \"kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.513692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.513559    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.513692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.513590    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.514280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.514257    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.516037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.516008    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.586851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.586792    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5kkd\" (UniqueName: \"kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd\") pod \"maas-api-revoke-sub-5f5149f8-2lhls\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:27.606637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:27.606613    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:28.011536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:28.011499    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"]
Nov 01 22:38:28.015633 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:38:28.015601    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41324245_36a3_40d7_a6c8_48d60d77bacb.slice/crio-45f16b85bc591f49992df03f8323bc1b6640132dab59bc05bf1913bac1fec9a9 WatchSource:0}: Error finding container 45f16b85bc591f49992df03f8323bc1b6640132dab59bc05bf1913bac1fec9a9: Status 404 returned error can't find the container with id 45f16b85bc591f49992df03f8323bc1b6640132dab59bc05bf1913bac1fec9a9
Nov 01 22:38:28.048190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:28.048033    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:38:28.048190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:28.048075    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 22:38:28.726569 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:28.726529    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls" event={"ID":"41324245-36a3-40d7-a6c8-48d60d77bacb","Type":"ContainerStarted","Data":"b79baee697971079b3dfc6b0bd235ab94d97ee8ab2f4fd3fbda464a4f22686e8"}
Nov 01 22:38:28.727037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:28.726597    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls" event={"ID":"41324245-36a3-40d7-a6c8-48d60d77bacb","Type":"ContainerStarted","Data":"45f16b85bc591f49992df03f8323bc1b6640132dab59bc05bf1913bac1fec9a9"}
Nov 01 22:38:28.878957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:28.878855    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls" podStartSLOduration=1.878837898 podStartE2EDuration="1.878837898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:28.874506388 +0000 UTC m=+890.192156373" watchObservedRunningTime="2026-11-01 22:38:28.878837898 +0000 UTC m=+890.196487926"
Nov 01 22:38:29.048460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.048428    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:38:29.048602 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.048475    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 22:38:29.292751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.291808    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"]
Nov 01 22:38:29.296870 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.296842    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.334367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.334333    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.334508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.334462    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.334560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.334514    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.334637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.334614    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq5b9\" (UniqueName: \"kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.334699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.334688    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.334808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.334716    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.347934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.347911    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"default-dockercfg-qgkfn\""
Nov 01 22:38:29.348116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.348008    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"premium-w0-c54de0-kserve-self-signed-certs\""
Nov 01 22:38:29.348260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.348132    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"kube-root-ca.crt\""
Nov 01 22:38:29.348353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.348214    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"openshift-service-ca.crt\""
Nov 01 22:38:29.348945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.348924    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"]
Nov 01 22:38:29.435907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.435830    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.435907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.435904    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.435940    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.436001    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq5b9\" (UniqueName: \"kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.436055    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.436077    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.436368    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.436422    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.436615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.436593    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.438460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.438438    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.438572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.438458    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.491746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.491255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:29.499760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.497222    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq5b9\" (UniqueName: \"kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9\") pod \"premium-w0-c54de0-kserve-5669f65b57-9px6b\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") " pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.612428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.612349    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:29.741666 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.741629    4531 generic.go:356] "Generic (PLEG): container finished" podID="41324245-36a3-40d7-a6c8-48d60d77bacb" containerID="b79baee697971079b3dfc6b0bd235ab94d97ee8ab2f4fd3fbda464a4f22686e8" exitCode=0
Nov 01 22:38:29.742118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:29.741813    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls" event={"ID":"41324245-36a3-40d7-a6c8-48d60d77bacb","Type":"ContainerDied","Data":"b79baee697971079b3dfc6b0bd235ab94d97ee8ab2f4fd3fbda464a4f22686e8"}
Nov 01 22:38:30.047607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.047522    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:38:30.047607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.047573    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 22:38:30.099015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.093790    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"]
Nov 01 22:38:30.099015 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:38:30.098953    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod364690f7_d8dc_4592_91cd_26fa27bf3aad.slice/crio-3f876dc930a90b83af7c2342cdc26f0c0289213b206ec474c0753af53e3a55b2 WatchSource:0}: Error finding container 3f876dc930a90b83af7c2342cdc26f0c0289213b206ec474c0753af53e3a55b2: Status 404 returned error can't find the container with id 3f876dc930a90b83af7c2342cdc26f0c0289213b206ec474c0753af53e3a55b2
Nov 01 22:38:30.751425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.751385    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" event={"ID":"364690f7-d8dc-4592-91cd-26fa27bf3aad","Type":"ContainerStarted","Data":"7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef"}
Nov 01 22:38:30.751868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.751437    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" event={"ID":"364690f7-d8dc-4592-91cd-26fa27bf3aad","Type":"ContainerStarted","Data":"3f876dc930a90b83af7c2342cdc26f0c0289213b206ec474c0753af53e3a55b2"}
Nov 01 22:38:30.868965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.868905    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podStartSLOduration=1.868885693 podStartE2EDuration="1.868885693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:38:30.863216387 +0000 UTC m=+892.180866388" watchObservedRunningTime="2026-11-01 22:38:30.868885693 +0000 UTC m=+892.186535695"
Nov 01 22:38:30.981781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:30.974216    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:31.047759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.047657    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body=
Nov 01 22:38:31.047759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.047702    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused"
Nov 01 22:38:31.051087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.051063    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca\") pod \"41324245-36a3-40d7-a6c8-48d60d77bacb\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") "
Nov 01 22:38:31.051236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.051160    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd\" (UniqueName: \"kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd\") pod \"41324245-36a3-40d7-a6c8-48d60d77bacb\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") "
Nov 01 22:38:31.051236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.051208    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca\") pod \"41324245-36a3-40d7-a6c8-48d60d77bacb\" (UID: \"41324245-36a3-40d7-a6c8-48d60d77bacb\") "
Nov 01 22:38:31.051599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.051571    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca" pod "41324245-36a3-40d7-a6c8-48d60d77bacb" (UID: "41324245-36a3-40d7-a6c8-48d60d77bacb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:38:31.053236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.053207    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca" pod "41324245-36a3-40d7-a6c8-48d60d77bacb" (UID: "41324245-36a3-40d7-a6c8-48d60d77bacb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:38:31.053415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.053391    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd" pod "41324245-36a3-40d7-a6c8-48d60d77bacb" (UID: "41324245-36a3-40d7-a6c8-48d60d77bacb"). InnerVolumeSpecName "kube-api-access-d5kkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:38:31.152750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.152703    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5kkd\" (UniqueName: \"kubernetes.io/projected/41324245-36a3-40d7-a6c8-48d60d77bacb-kube-api-access-d5kkd\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:31.152750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.152750    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:31.152925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.152762    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41324245-36a3-40d7-a6c8-48d60d77bacb-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:31.320064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.319978    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:31.613153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.613076    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:31.760139 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.760108    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"
Nov 01 22:38:31.760139 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.760120    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls" event={"ID":"41324245-36a3-40d7-a6c8-48d60d77bacb","Type":"ContainerDied","Data":"45f16b85bc591f49992df03f8323bc1b6640132dab59bc05bf1913bac1fec9a9"}
Nov 01 22:38:31.760584 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:31.760166    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45f16b85bc591f49992df03f8323bc1b6640132dab59bc05bf1913bac1fec9a9"
Nov 01 22:38:32.242498 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:32.242462    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:32.243764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:32.243017    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:32.244339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:32.244307    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 22:38:32.267528 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:32.267496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:32.471435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:32.471353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:36.247624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:36.247590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:38.037804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.037773    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"]
Nov 01 22:38:38.038365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.038350    4531 cpu_manager.go:414] "containerMap: removing container" podUID="41324245-36a3-40d7-a6c8-48d60d77bacb" containerName="revoke-keys"
Nov 01 22:38:38.038424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.038366    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41324245-36a3-40d7-a6c8-48d60d77bacb" containerName="revoke-keys"
Nov 01 22:38:38.038463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.038454    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="41324245-36a3-40d7-a6c8-48d60d77bacb" containerName="revoke-keys"
Nov 01 22:38:38.042018 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.041997    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.094132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.094107    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:38:38.125239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.125211    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvh9\" (UniqueName: \"kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.125364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.125251    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.125493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.125466    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.140117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.140096    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:38:38.143733 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.143706    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"]
Nov 01 22:38:38.225990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.225960    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.226110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.226028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvh9\" (UniqueName: \"kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.226110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.226051    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.226741 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.226704    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.228242 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.228223    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.280945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.280922    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvh9\" (UniqueName: \"kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9\") pod \"maas-api-revoke-sub-de97764e-l5df8\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.355236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.355188    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:38.472110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.472063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:38.731011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.730983    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"]
Nov 01 22:38:38.734848 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:38:38.734817    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dd9534_ac50_43fd_bd9f_cb2be30c2e20.slice/crio-cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa WatchSource:0}: Error finding container cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa: Status 404 returned error can't find the container with id cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa
Nov 01 22:38:38.807776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:38.807752    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8" event={"ID":"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20","Type":"ContainerStarted","Data":"cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa"}
Nov 01 22:38:39.612545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:39.612515    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:39.624873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:39.624843    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:39.816379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:39.816347    4531 generic.go:356] "Generic (PLEG): container finished" podID="e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" containerID="ba64f6acfbd3d2c814f1006bd0c8ac9f200036e85ff4e18aad9d10e8fdd9534e" exitCode=0
Nov 01 22:38:39.816541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:39.816438    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8" event={"ID":"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20","Type":"ContainerDied","Data":"ba64f6acfbd3d2c814f1006bd0c8ac9f200036e85ff4e18aad9d10e8fdd9534e"}
Nov 01 22:38:39.827832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:39.827806    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:38:40.985100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:40.985069    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:41.059609 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.059577    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca\") pod \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") "
Nov 01 22:38:41.059810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.059638    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9\" (UniqueName: \"kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9\") pod \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") "
Nov 01 22:38:41.059810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.059691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca\") pod \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\" (UID: \"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20\") "
Nov 01 22:38:41.059947 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.059926    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca" pod "e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" (UID: "e0dd9534-ac50-43fd-bd9f-cb2be30c2e20"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:38:41.060078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.060057    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:41.061894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.061867    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca" pod "e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" (UID: "e0dd9534-ac50-43fd-bd9f-cb2be30c2e20"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:38:41.062008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.061916    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9" pod "e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" (UID: "e0dd9534-ac50-43fd-bd9f-cb2be30c2e20"). InnerVolumeSpecName "kube-api-access-gxvh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:38:41.064699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.064678    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:41.161337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.161272    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxvh9\" (UniqueName: \"kubernetes.io/projected/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-kube-api-access-gxvh9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:41.161337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.161295    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:38:41.832496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.832465    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"
Nov 01 22:38:41.832821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.832501    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8" event={"ID":"e0dd9534-ac50-43fd-bd9f-cb2be30c2e20","Type":"ContainerDied","Data":"cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa"}
Nov 01 22:38:41.832821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:41.832550    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa"
Nov 01 22:38:41.888006 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:38:41.885653    4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dd9534_ac50_43fd_bd9f_cb2be30c2e20.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dd9534_ac50_43fd_bd9f_cb2be30c2e20.slice/crio-cba2e6e71edb53aabf271bacb0713ebc8b4ae90b62be1569a925e3e340b591aa\": RecentStats: unable to find data in memory cache]"
Nov 01 22:38:42.526426 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:38:42.526388    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:38:45.932873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:45.932840    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:47.949422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:47.949374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:50.831643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:50.831608    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:55.441364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:55.441331    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:55.638894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:55.638857    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:38:55.846194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:38:55.846162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:00.551746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:00.551694    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:05.497719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:05.497672    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:10.271094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:10.271017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:12.526658 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:39:12.526618    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:39:15.149400 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:15.149366    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:20.125471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:20.125387    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:24.133801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.133761    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"]
Nov 01 22:39:24.134318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.134302    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" containerName="revoke-keys"
Nov 01 22:39:24.134364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.134319    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" containerName="revoke-keys"
Nov 01 22:39:24.134435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.134426    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" containerName="revoke-keys"
Nov 01 22:39:24.137924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.137908    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.234630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.234596    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:24.234630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.234597    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:24.238325 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.238296    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"]
Nov 01 22:39:24.266624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.266588    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.266771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.266652    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.266867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.266841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsgcn\" (UniqueName: \"kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.368446 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.368417    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsgcn\" (UniqueName: \"kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.368592 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.368558    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.368678 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.368596    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.369494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.369466    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.371135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.371113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.422062 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.422002    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsgcn\" (UniqueName: \"kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn\") pod \"maas-api-revoke-sub-2310dbb2-wv8ps\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.451480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.451460    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:24.846902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:24.846875    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"]
Nov 01 22:39:24.850589 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:24.850562    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e3b5c4_fa38_45b1_9861_9ce4dfbecccf.slice/crio-3333ca55598b4f1391a9cb47042dff4a2c02851b2fa57c1405d7daa112e36a0b WatchSource:0}: Error finding container 3333ca55598b4f1391a9cb47042dff4a2c02851b2fa57c1405d7daa112e36a0b: Status 404 returned error can't find the container with id 3333ca55598b4f1391a9cb47042dff4a2c02851b2fa57c1405d7daa112e36a0b
Nov 01 22:39:25.044333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:25.044297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:25.214042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:25.214005    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps" event={"ID":"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf","Type":"ContainerStarted","Data":"3333ca55598b4f1391a9cb47042dff4a2c02851b2fa57c1405d7daa112e36a0b"}
Nov 01 22:39:26.222614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.222581    4531 generic.go:356] "Generic (PLEG): container finished" podID="71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" containerID="fa385500b44e94b026dd2e9b62ab30f6d2671418d5c3f833e2356796d6e30a44" exitCode=0
Nov 01 22:39:26.223064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.222656    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps" event={"ID":"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf","Type":"ContainerDied","Data":"fa385500b44e94b026dd2e9b62ab30f6d2671418d5c3f833e2356796d6e30a44"}
Nov 01 22:39:26.849391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.849357    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"]
Nov 01 22:39:26.853447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.853425    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:26.907912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.907886    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"]
Nov 01 22:39:26.995414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.995368    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:26.995620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.995596    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:26.995746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:26.995708    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjhst\" (UniqueName: \"kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.097238 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.097205    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.097396 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.097258    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjhst\" (UniqueName: \"kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.097396 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.097334    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.097923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.097898    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.099650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.099596    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.150800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.150773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjhst\" (UniqueName: \"kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst\") pod \"maas-api-revoke-sub-319bbba5-hf7j4\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.167231 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.167196    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:27.397280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.397258    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:27.501500 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.501470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn\" (UniqueName: \"kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn\") pod \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") "
Nov 01 22:39:27.501641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.501577    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca\") pod \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") "
Nov 01 22:39:27.501693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.501676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca\") pod \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\" (UID: \"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf\") "
Nov 01 22:39:27.502089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.502059    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca" pod "71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" (UID: "71e3b5c4-fa38-45b1-9861-9ce4dfbecccf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:27.503539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.503509    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn" pod "71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" (UID: "71e3b5c4-fa38-45b1-9861-9ce4dfbecccf"). InnerVolumeSpecName "kube-api-access-nsgcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:27.503677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.503613    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca" pod "71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" (UID: "71e3b5c4-fa38-45b1-9861-9ce4dfbecccf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:27.550223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.550194    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"]
Nov 01 22:39:27.594770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:27.594741    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a86243_8729_496f_a291_95b8d37ff93a.slice/crio-944d01274161323387ca49a849ae11cacecaa051adeaf0ebca3ccf4c5eb8fd0c WatchSource:0}: Error finding container 944d01274161323387ca49a849ae11cacecaa051adeaf0ebca3ccf4c5eb8fd0c: Status 404 returned error can't find the container with id 944d01274161323387ca49a849ae11cacecaa051adeaf0ebca3ccf4c5eb8fd0c
Nov 01 22:39:27.603243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.603214    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsgcn\" (UniqueName: \"kubernetes.io/projected/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-kube-api-access-nsgcn\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:27.603363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.603247    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:27.603363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:27.603262    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:28.239977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:28.239953    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"
Nov 01 22:39:28.240121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:28.239994    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps" event={"ID":"71e3b5c4-fa38-45b1-9861-9ce4dfbecccf","Type":"ContainerDied","Data":"3333ca55598b4f1391a9cb47042dff4a2c02851b2fa57c1405d7daa112e36a0b"}
Nov 01 22:39:28.240121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:28.240046    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3333ca55598b4f1391a9cb47042dff4a2c02851b2fa57c1405d7daa112e36a0b"
Nov 01 22:39:28.241758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:28.241716    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4" event={"ID":"c3a86243-8729-496f-a291-95b8d37ff93a","Type":"ContainerStarted","Data":"09bb93e83ff85a27db7ee909716d3ddff4dd30680f9c932a4ed428c0dfaf4782"}
Nov 01 22:39:28.241896 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:28.241767    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4" event={"ID":"c3a86243-8729-496f-a291-95b8d37ff93a","Type":"ContainerStarted","Data":"944d01274161323387ca49a849ae11cacecaa051adeaf0ebca3ccf4c5eb8fd0c"}
Nov 01 22:39:28.352614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:28.352563    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4" podStartSLOduration=2.35254945 podStartE2EDuration="2.35254945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:28.349083165 +0000 UTC m=+949.666733178" watchObservedRunningTime="2026-11-01 22:39:28.35254945 +0000 UTC m=+949.670199451"
Nov 01 22:39:29.251092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:29.251051    4531 generic.go:356] "Generic (PLEG): container finished" podID="c3a86243-8729-496f-a291-95b8d37ff93a" containerID="09bb93e83ff85a27db7ee909716d3ddff4dd30680f9c932a4ed428c0dfaf4782" exitCode=0
Nov 01 22:39:29.251540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:29.251089    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4" event={"ID":"c3a86243-8729-496f-a291-95b8d37ff93a","Type":"ContainerDied","Data":"09bb93e83ff85a27db7ee909716d3ddff4dd30680f9c932a4ed428c0dfaf4782"}
Nov 01 22:39:29.739522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:29.739486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:30.396558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.396534    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:30.531536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.531461    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst\" (UniqueName: \"kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst\") pod \"c3a86243-8729-496f-a291-95b8d37ff93a\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") "
Nov 01 22:39:30.531711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.531602    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca\") pod \"c3a86243-8729-496f-a291-95b8d37ff93a\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") "
Nov 01 22:39:30.531711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.531679    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca\") pod \"c3a86243-8729-496f-a291-95b8d37ff93a\" (UID: \"c3a86243-8729-496f-a291-95b8d37ff93a\") "
Nov 01 22:39:30.532052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.532015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca" pod "c3a86243-8729-496f-a291-95b8d37ff93a" (UID: "c3a86243-8729-496f-a291-95b8d37ff93a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:30.532140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.532123    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.533847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.533824    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst" pod "c3a86243-8729-496f-a291-95b8d37ff93a" (UID: "c3a86243-8729-496f-a291-95b8d37ff93a"). InnerVolumeSpecName "kube-api-access-fjhst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:30.533967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.533880    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca" pod "c3a86243-8729-496f-a291-95b8d37ff93a" (UID: "c3a86243-8729-496f-a291-95b8d37ff93a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:30.633188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.633157    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a86243-8729-496f-a291-95b8d37ff93a-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:30.633188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:30.633181    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjhst\" (UniqueName: \"kubernetes.io/projected/c3a86243-8729-496f-a291-95b8d37ff93a-kube-api-access-fjhst\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:31.267683 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:31.267657    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"
Nov 01 22:39:31.267875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:31.267656    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4" event={"ID":"c3a86243-8729-496f-a291-95b8d37ff93a","Type":"ContainerDied","Data":"944d01274161323387ca49a849ae11cacecaa051adeaf0ebca3ccf4c5eb8fd0c"}
Nov 01 22:39:31.267875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:31.267794    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944d01274161323387ca49a849ae11cacecaa051adeaf0ebca3ccf4c5eb8fd0c"
Nov 01 22:39:32.795469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:32.795433    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:34.798594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:34.798562    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:39.755014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:39.754967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:40.892241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892213    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"]
Nov 01 22:39:40.892765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892750    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c3a86243-8729-496f-a291-95b8d37ff93a" containerName="revoke-keys"
Nov 01 22:39:40.892814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892766    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a86243-8729-496f-a291-95b8d37ff93a" containerName="revoke-keys"
Nov 01 22:39:40.892814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892789    4531 cpu_manager.go:414] "containerMap: removing container" podUID="71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" containerName="revoke-keys"
Nov 01 22:39:40.892814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892794    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" containerName="revoke-keys"
Nov 01 22:39:40.892907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892898    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a86243-8729-496f-a291-95b8d37ff93a" containerName="revoke-keys"
Nov 01 22:39:40.892939 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.892908    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" containerName="revoke-keys"
Nov 01 22:39:40.896925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.896910    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:40.991949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.991906    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:40.991949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.991922    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:40.997789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:40.997751    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"]
Nov 01 22:39:41.062216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.062193    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x565q\" (UniqueName: \"kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.062334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.062231    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.062421 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.062402    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.163268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.163198    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.163268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.163249    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x565q\" (UniqueName: \"kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.163433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.163368    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.164219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.164191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.165568 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.165536    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.222363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.222335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x565q\" (UniqueName: \"kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q\") pod \"maas-api-revoke-sub-d212ba5e-fhnv7\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.510173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.510142    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:41.894788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:41.894702    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"]
Nov 01 22:39:41.899141 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:41.899094    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e75d00a_9ede_44af_b2f9_74e0d66bf058.slice/crio-6850070ceeeb7948cb56edcb6beab44797dc9ada53d4031635c196c231a62f2f WatchSource:0}: Error finding container 6850070ceeeb7948cb56edcb6beab44797dc9ada53d4031635c196c231a62f2f: Status 404 returned error can't find the container with id 6850070ceeeb7948cb56edcb6beab44797dc9ada53d4031635c196c231a62f2f
Nov 01 22:39:42.413938 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:42.413900    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7" event={"ID":"8e75d00a-9ede-44af-b2f9-74e0d66bf058","Type":"ContainerStarted","Data":"095e5358311020ba56dea06ac4389b7a8ec01e48e553456423279d7c639cf90d"}
Nov 01 22:39:42.414114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:42.413980    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7" event={"ID":"8e75d00a-9ede-44af-b2f9-74e0d66bf058","Type":"ContainerStarted","Data":"6850070ceeeb7948cb56edcb6beab44797dc9ada53d4031635c196c231a62f2f"}
Nov 01 22:39:42.522422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:42.522371    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7" podStartSLOduration=2.522353185 podStartE2EDuration="2.522353185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:42.520971286 +0000 UTC m=+963.838621282" watchObservedRunningTime="2026-11-01 22:39:42.522353185 +0000 UTC m=+963.840003187"
Nov 01 22:39:42.528232 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:39:42.528202    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:39:43.422059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:43.422027    4531 generic.go:356] "Generic (PLEG): container finished" podID="8e75d00a-9ede-44af-b2f9-74e0d66bf058" containerID="095e5358311020ba56dea06ac4389b7a8ec01e48e553456423279d7c639cf90d" exitCode=0
Nov 01 22:39:43.422451 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:43.422116    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7" event={"ID":"8e75d00a-9ede-44af-b2f9-74e0d66bf058","Type":"ContainerDied","Data":"095e5358311020ba56dea06ac4389b7a8ec01e48e553456423279d7c639cf90d"}
Nov 01 22:39:44.131959 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.131925    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"]
Nov 01 22:39:44.136592 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.136572    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.190553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.190528    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"]
Nov 01 22:39:44.296222 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.296192    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.296371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.296259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n78fn\" (UniqueName: \"kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.296465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.296438    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.397688 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.397628    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.397830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.397695    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.397830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.397770    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n78fn\" (UniqueName: \"kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.398346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.398326    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.399762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.399744    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.452415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.452390    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n78fn\" (UniqueName: \"kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn\") pod \"maas-api-revoke-sub-4b0f084d-klgl9\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.453118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.453100    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:44.535412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.535275    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:44.586068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.586044    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:44.706786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.701158    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q\" (UniqueName: \"kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q\") pod \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") "
Nov 01 22:39:44.706786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.701229    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca\") pod \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") "
Nov 01 22:39:44.706786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.701303    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca\") pod \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\" (UID: \"8e75d00a-9ede-44af-b2f9-74e0d66bf058\") "
Nov 01 22:39:44.706786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.702101    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca" pod "8e75d00a-9ede-44af-b2f9-74e0d66bf058" (UID: "8e75d00a-9ede-44af-b2f9-74e0d66bf058"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:44.706786 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.705314    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca" pod "8e75d00a-9ede-44af-b2f9-74e0d66bf058" (UID: "8e75d00a-9ede-44af-b2f9-74e0d66bf058"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:44.707206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.707185    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q" pod "8e75d00a-9ede-44af-b2f9-74e0d66bf058" (UID: "8e75d00a-9ede-44af-b2f9-74e0d66bf058"). InnerVolumeSpecName "kube-api-access-x565q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:44.802766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.802721    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x565q\" (UniqueName: \"kubernetes.io/projected/8e75d00a-9ede-44af-b2f9-74e0d66bf058-kube-api-access-x565q\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:44.802766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.802767    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:44.802944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.802785    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e75d00a-9ede-44af-b2f9-74e0d66bf058-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:44.845091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:44.845056    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"]
Nov 01 22:39:44.849528 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:44.849506    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d2b247_7e69_4ba1_b043_496b56110ab5.slice/crio-7222b93b3a950a2f59a607175279353db75acad9e60f8a997160a85406886c5c WatchSource:0}: Error finding container 7222b93b3a950a2f59a607175279353db75acad9e60f8a997160a85406886c5c: Status 404 returned error can't find the container with id 7222b93b3a950a2f59a607175279353db75acad9e60f8a997160a85406886c5c
Nov 01 22:39:45.437542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:45.437516    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"
Nov 01 22:39:45.437868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:45.437557    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7" event={"ID":"8e75d00a-9ede-44af-b2f9-74e0d66bf058","Type":"ContainerDied","Data":"6850070ceeeb7948cb56edcb6beab44797dc9ada53d4031635c196c231a62f2f"}
Nov 01 22:39:45.437868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:45.437617    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6850070ceeeb7948cb56edcb6beab44797dc9ada53d4031635c196c231a62f2f"
Nov 01 22:39:45.439320 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:45.439285    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9" event={"ID":"44d2b247-7e69-4ba1-b043-496b56110ab5","Type":"ContainerStarted","Data":"feb6bcbfaa9a503671f232f11f8476ca4fb8eef94bfc6e084bbe194f01c76f00"}
Nov 01 22:39:45.439460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:45.439338    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9" event={"ID":"44d2b247-7e69-4ba1-b043-496b56110ab5","Type":"ContainerStarted","Data":"7222b93b3a950a2f59a607175279353db75acad9e60f8a997160a85406886c5c"}
Nov 01 22:39:45.547666 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:45.547619    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9" podStartSLOduration=1.547605165 podStartE2EDuration="1.547605165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:45.546680171 +0000 UTC m=+966.864330175" watchObservedRunningTime="2026-11-01 22:39:45.547605165 +0000 UTC m=+966.865255190"
Nov 01 22:39:46.448483 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:46.448450    4531 generic.go:356] "Generic (PLEG): container finished" podID="44d2b247-7e69-4ba1-b043-496b56110ab5" containerID="feb6bcbfaa9a503671f232f11f8476ca4fb8eef94bfc6e084bbe194f01c76f00" exitCode=0
Nov 01 22:39:46.448757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:46.448540    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9" event={"ID":"44d2b247-7e69-4ba1-b043-496b56110ab5","Type":"ContainerDied","Data":"feb6bcbfaa9a503671f232f11f8476ca4fb8eef94bfc6e084bbe194f01c76f00"}
Nov 01 22:39:47.615064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.615032    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:47.731314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.731247    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca\") pod \"44d2b247-7e69-4ba1-b043-496b56110ab5\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") "
Nov 01 22:39:47.731314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.731296    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca\") pod \"44d2b247-7e69-4ba1-b043-496b56110ab5\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") "
Nov 01 22:39:47.731527 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.731342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn\" (UniqueName: \"kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn\") pod \"44d2b247-7e69-4ba1-b043-496b56110ab5\" (UID: \"44d2b247-7e69-4ba1-b043-496b56110ab5\") "
Nov 01 22:39:47.731606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.731583    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca" pod "44d2b247-7e69-4ba1-b043-496b56110ab5" (UID: "44d2b247-7e69-4ba1-b043-496b56110ab5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:47.731771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.731751    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:47.734062 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.734037    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca" pod "44d2b247-7e69-4ba1-b043-496b56110ab5" (UID: "44d2b247-7e69-4ba1-b043-496b56110ab5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:47.734263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.734235    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn" pod "44d2b247-7e69-4ba1-b043-496b56110ab5" (UID: "44d2b247-7e69-4ba1-b043-496b56110ab5"). InnerVolumeSpecName "kube-api-access-n78fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:47.833311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.833284    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44d2b247-7e69-4ba1-b043-496b56110ab5-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:47.833311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:47.833308    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n78fn\" (UniqueName: \"kubernetes.io/projected/44d2b247-7e69-4ba1-b043-496b56110ab5-kube-api-access-n78fn\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:48.464105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:48.464075    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"
Nov 01 22:39:48.464274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:48.464113    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9" event={"ID":"44d2b247-7e69-4ba1-b043-496b56110ab5","Type":"ContainerDied","Data":"7222b93b3a950a2f59a607175279353db75acad9e60f8a997160a85406886c5c"}
Nov 01 22:39:48.464274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:48.464160    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7222b93b3a950a2f59a607175279353db75acad9e60f8a997160a85406886c5c"
Nov 01 22:39:49.392002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:49.391967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:52.034505 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.034472    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"]
Nov 01 22:39:52.035095 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.035078    4531 cpu_manager.go:414] "containerMap: removing container" podUID="44d2b247-7e69-4ba1-b043-496b56110ab5" containerName="revoke-keys"
Nov 01 22:39:52.035143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.035096    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d2b247-7e69-4ba1-b043-496b56110ab5" containerName="revoke-keys"
Nov 01 22:39:52.035143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.035122    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e75d00a-9ede-44af-b2f9-74e0d66bf058" containerName="revoke-keys"
Nov 01 22:39:52.035143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.035128    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e75d00a-9ede-44af-b2f9-74e0d66bf058" containerName="revoke-keys"
Nov 01 22:39:52.035240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.035211    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d2b247-7e69-4ba1-b043-496b56110ab5" containerName="revoke-keys"
Nov 01 22:39:52.035240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.035222    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e75d00a-9ede-44af-b2f9-74e0d66bf058" containerName="revoke-keys"
Nov 01 22:39:52.038704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.038685    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.084026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.083991    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.084169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.084103    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzf4s\" (UniqueName: \"kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.084169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.084147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.088470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.088445    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:52.088644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.088486    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:52.095400 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.095379    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"]
Nov 01 22:39:52.184843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.184811    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.184985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.184924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzf4s\" (UniqueName: \"kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.184985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.184965    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.185773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.185716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.187159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.187136    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.238133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.238109    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzf4s\" (UniqueName: \"kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s\") pod \"maas-api-revoke-sub-886c94c7-xkt4f\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.351431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.351365    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:52.747493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:52.747462    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"]
Nov 01 22:39:52.751151 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:52.751123    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe2df68a_8101_491a_901e_6191f12ec192.slice/crio-b90198f03cce66fa9ba5bf7e3b1dfe6f667107a75d5a5aba76a307a8758cc24f WatchSource:0}: Error finding container b90198f03cce66fa9ba5bf7e3b1dfe6f667107a75d5a5aba76a307a8758cc24f: Status 404 returned error can't find the container with id b90198f03cce66fa9ba5bf7e3b1dfe6f667107a75d5a5aba76a307a8758cc24f
Nov 01 22:39:53.517046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:53.517005    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f" event={"ID":"be2df68a-8101-491a-901e-6191f12ec192","Type":"ContainerStarted","Data":"847fd533c15c33f8044894fe7d9d9ac6e92279b6e1e2db41588fe3f36a3df248"}
Nov 01 22:39:53.517554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:53.517063    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f" event={"ID":"be2df68a-8101-491a-901e-6191f12ec192","Type":"ContainerStarted","Data":"b90198f03cce66fa9ba5bf7e3b1dfe6f667107a75d5a5aba76a307a8758cc24f"}
Nov 01 22:39:53.626045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:53.625993    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f" podStartSLOduration=2.625980204 podStartE2EDuration="2.625980204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:53.624129945 +0000 UTC m=+974.941779972" watchObservedRunningTime="2026-11-01 22:39:53.625980204 +0000 UTC m=+974.943630204"
Nov 01 22:39:54.264008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:54.263973    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:54.525925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:54.525848    4531 generic.go:356] "Generic (PLEG): container finished" podID="be2df68a-8101-491a-901e-6191f12ec192" containerID="847fd533c15c33f8044894fe7d9d9ac6e92279b6e1e2db41588fe3f36a3df248" exitCode=0
Nov 01 22:39:54.525925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:54.525877    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f" event={"ID":"be2df68a-8101-491a-901e-6191f12ec192","Type":"ContainerDied","Data":"847fd533c15c33f8044894fe7d9d9ac6e92279b6e1e2db41588fe3f36a3df248"}
Nov 01 22:39:55.712523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.712502    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:55.829950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.829564    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s\" (UniqueName: \"kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s\") pod \"be2df68a-8101-491a-901e-6191f12ec192\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") "
Nov 01 22:39:55.829950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.829625    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca\") pod \"be2df68a-8101-491a-901e-6191f12ec192\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") "
Nov 01 22:39:55.830266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.830244    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca\") pod \"be2df68a-8101-491a-901e-6191f12ec192\" (UID: \"be2df68a-8101-491a-901e-6191f12ec192\") "
Nov 01 22:39:55.833392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.833358    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca" pod "be2df68a-8101-491a-901e-6191f12ec192" (UID: "be2df68a-8101-491a-901e-6191f12ec192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:39:55.833890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.833858    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca" pod "be2df68a-8101-491a-901e-6191f12ec192" (UID: "be2df68a-8101-491a-901e-6191f12ec192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:39:55.835077 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.835049    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s" pod "be2df68a-8101-491a-901e-6191f12ec192" (UID: "be2df68a-8101-491a-901e-6191f12ec192"). InnerVolumeSpecName "kube-api-access-vzf4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:39:55.932514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.932483    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzf4s\" (UniqueName: \"kubernetes.io/projected/be2df68a-8101-491a-901e-6191f12ec192-kube-api-access-vzf4s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:55.932514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.932508    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be2df68a-8101-491a-901e-6191f12ec192-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:55.932514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:55.932519    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be2df68a-8101-491a-901e-6191f12ec192-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:39:56.545737 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:56.545693    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"
Nov 01 22:39:56.545913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:56.545799    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f" event={"ID":"be2df68a-8101-491a-901e-6191f12ec192","Type":"ContainerDied","Data":"b90198f03cce66fa9ba5bf7e3b1dfe6f667107a75d5a5aba76a307a8758cc24f"}
Nov 01 22:39:56.545913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:56.545896    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90198f03cce66fa9ba5bf7e3b1dfe6f667107a75d5a5aba76a307a8758cc24f"
Nov 01 22:39:57.344863 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.344831    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"]
Nov 01 22:39:57.345391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.345375    4531 cpu_manager.go:414] "containerMap: removing container" podUID="be2df68a-8101-491a-901e-6191f12ec192" containerName="revoke-keys"
Nov 01 22:39:57.345436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.345393    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be2df68a-8101-491a-901e-6191f12ec192" containerName="revoke-keys"
Nov 01 22:39:57.345503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.345495    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="be2df68a-8101-491a-901e-6191f12ec192" containerName="revoke-keys"
Nov 01 22:39:57.349164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.349147    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.408135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.407721    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"]
Nov 01 22:39:57.446515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.446482    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:39:57.446688 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.446592    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:39:57.450526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.450508    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.450668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.450650    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.450713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.450678    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48xvx\" (UniqueName: \"kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.552037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.552012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.552037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.552038    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48xvx\" (UniqueName: \"kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.552242 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.552101    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.552803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.552783    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.554366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.554346    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.608267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.608202    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48xvx\" (UniqueName: \"kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx\") pod \"maas-api-revoke-sub-00f5df69-dzszf\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:57.672031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:57.671999    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:39:58.081261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:58.081219    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"]
Nov 01 22:39:58.100714 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:58.087418    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aaab0b9_11e2_4f72_9353_0854513d01f8.slice/crio-c09e3eb2e7c10c67f2c7e7a2b826a7bea1dd028953bfa283ccf724f92d174a40 WatchSource:0}: Error finding container c09e3eb2e7c10c67f2c7e7a2b826a7bea1dd028953bfa283ccf724f92d174a40: Status 404 returned error can't find the container with id c09e3eb2e7c10c67f2c7e7a2b826a7bea1dd028953bfa283ccf724f92d174a40
Nov 01 22:39:58.562806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:58.562769    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf" event={"ID":"2aaab0b9-11e2-4f72-9353-0854513d01f8","Type":"ContainerStarted","Data":"4c990c246ce477b523f899dffdd13e0ebb75bd670daa2808806c40800749b547"}
Nov 01 22:39:58.563249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:58.562830    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf" event={"ID":"2aaab0b9-11e2-4f72-9353-0854513d01f8","Type":"ContainerStarted","Data":"c09e3eb2e7c10c67f2c7e7a2b826a7bea1dd028953bfa283ccf724f92d174a40"}
Nov 01 22:39:58.674111 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:58.674028    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf" podStartSLOduration=1.6740098369999998 podStartE2EDuration="1.674009837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:39:58.67321308 +0000 UTC m=+979.990863093" watchObservedRunningTime="2026-11-01 22:39:58.674009837 +0000 UTC m=+979.991659841"
Nov 01 22:39:59.163736 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.163693    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"]
Nov 01 22:39:59.168772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.168750    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.219801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.219774    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-d6101b\"/\"distinct-w5-d6101b-kserve-self-signed-certs\""
Nov 01 22:39:59.269953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.269918    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.270231 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.270203    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.270476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.270446    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.270681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.270664    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm4xx\" (UniqueName: \"kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.270893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.270868    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.270970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.270906    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.271104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.271085    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"]
Nov 01 22:39:59.315501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.315472    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:59.373209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.372964    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.373209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.373051    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm4xx\" (UniqueName: \"kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.373209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.373081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.373209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.373107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.373209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.373155    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.373519 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.373214    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.373773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.373653    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.375003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.374977    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.380161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.374354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.380742 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.380704    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.391404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.391378    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.428699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.428663    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm4xx\" (UniqueName: \"kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx\") pod \"distinct-w5-d6101b-kserve-c946db95b-r9m8r\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") " pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.496071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.496031    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:39:59.533903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.533853    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:39:59.584826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.584045    4531 generic.go:356] "Generic (PLEG): container finished" podID="2aaab0b9-11e2-4f72-9353-0854513d01f8" containerID="4c990c246ce477b523f899dffdd13e0ebb75bd670daa2808806c40800749b547" exitCode=0
Nov 01 22:39:59.584826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.584191    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf" event={"ID":"2aaab0b9-11e2-4f72-9353-0854513d01f8","Type":"ContainerDied","Data":"4c990c246ce477b523f899dffdd13e0ebb75bd670daa2808806c40800749b547"}
Nov 01 22:39:59.894781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:39:59.894603    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"]
Nov 01 22:39:59.897855 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:39:59.897827    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod996218de_84d9_4c3f_a3b5_4a09e7109fe5.slice/crio-60e56c78d8b1c9d51d5dec833514423a2ad67a533c03eb36b875bf7362be7519 WatchSource:0}: Error finding container 60e56c78d8b1c9d51d5dec833514423a2ad67a533c03eb36b875bf7362be7519: Status 404 returned error can't find the container with id 60e56c78d8b1c9d51d5dec833514423a2ad67a533c03eb36b875bf7362be7519
Nov 01 22:40:00.594166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.594124    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" event={"ID":"996218de-84d9-4c3f-a3b5-4a09e7109fe5","Type":"ContainerStarted","Data":"05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde"}
Nov 01 22:40:00.594642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.594181    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" event={"ID":"996218de-84d9-4c3f-a3b5-4a09e7109fe5","Type":"ContainerStarted","Data":"60e56c78d8b1c9d51d5dec833514423a2ad67a533c03eb36b875bf7362be7519"}
Nov 01 22:40:00.708228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.708174    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podStartSLOduration=1.708158846 podStartE2EDuration="1.708158846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:00.703855946 +0000 UTC m=+982.021505947" watchObservedRunningTime="2026-11-01 22:40:00.708158846 +0000 UTC m=+982.025808848"
Nov 01 22:40:00.747484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.747462    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:40:00.788657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.788631    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca\") pod \"2aaab0b9-11e2-4f72-9353-0854513d01f8\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") "
Nov 01 22:40:00.788819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.788799    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx\" (UniqueName: \"kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx\") pod \"2aaab0b9-11e2-4f72-9353-0854513d01f8\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") "
Nov 01 22:40:00.788928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.788913    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca\") pod \"2aaab0b9-11e2-4f72-9353-0854513d01f8\" (UID: \"2aaab0b9-11e2-4f72-9353-0854513d01f8\") "
Nov 01 22:40:00.789052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.789024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca" pod "2aaab0b9-11e2-4f72-9353-0854513d01f8" (UID: "2aaab0b9-11e2-4f72-9353-0854513d01f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:00.789253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.789237    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:00.790884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.790855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx" pod "2aaab0b9-11e2-4f72-9353-0854513d01f8" (UID: "2aaab0b9-11e2-4f72-9353-0854513d01f8"). InnerVolumeSpecName "kube-api-access-48xvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:00.790966 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.790896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca" pod "2aaab0b9-11e2-4f72-9353-0854513d01f8" (UID: "2aaab0b9-11e2-4f72-9353-0854513d01f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:00.892395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.891195    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2aaab0b9-11e2-4f72-9353-0854513d01f8-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:00.892395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:00.891230    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48xvx\" (UniqueName: \"kubernetes.io/projected/2aaab0b9-11e2-4f72-9353-0854513d01f8-kube-api-access-48xvx\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:01.496323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:01.496288    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:40:01.603628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:01.603599    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"
Nov 01 22:40:01.604151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:01.603635    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf" event={"ID":"2aaab0b9-11e2-4f72-9353-0854513d01f8","Type":"ContainerDied","Data":"c09e3eb2e7c10c67f2c7e7a2b826a7bea1dd028953bfa283ccf724f92d174a40"}
Nov 01 22:40:01.604151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:01.603696    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c09e3eb2e7c10c67f2c7e7a2b826a7bea1dd028953bfa283ccf724f92d174a40"
Nov 01 22:40:04.047930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:04.047895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:09.118550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.118508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:09.159450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.159416    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"]
Nov 01 22:40:09.160020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.160003    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2aaab0b9-11e2-4f72-9353-0854513d01f8" containerName="revoke-keys"
Nov 01 22:40:09.160020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.160021    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aaab0b9-11e2-4f72-9353-0854513d01f8" containerName="revoke-keys"
Nov 01 22:40:09.160143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.160134    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aaab0b9-11e2-4f72-9353-0854513d01f8" containerName="revoke-keys"
Nov 01 22:40:09.163715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.163699    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.213126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.213105    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:40:09.259578 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.259549    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:40:09.265181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.265155    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"]
Nov 01 22:40:09.279436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.279407    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.279547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.279530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.279610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.279598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7wtv\" (UniqueName: \"kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.380510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.380438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.380510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.380509    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.380683 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.380558    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7wtv\" (UniqueName: \"kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.381238 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.381210    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.382680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.382655    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.435256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.435230    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7wtv\" (UniqueName: \"kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv\") pod \"maas-api-revoke-sub-7a9f237a-7m6hm\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.478645 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.478620    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:09.496822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.496798    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:40:09.513722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.513698    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:40:09.680776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.680693    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:40:09.863372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:09.863343    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"]
Nov 01 22:40:09.867598 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:09.867570    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod411cba67_a78b_4479_98e8_588fc9a4e2ce.slice/crio-ccb2438686cfba948cd91917002440591f8140df4114f215384306b53100df00 WatchSource:0}: Error finding container ccb2438686cfba948cd91917002440591f8140df4114f215384306b53100df00: Status 404 returned error can't find the container with id ccb2438686cfba948cd91917002440591f8140df4114f215384306b53100df00
Nov 01 22:40:10.679314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:10.679282    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm" event={"ID":"411cba67-a78b-4479-98e8-588fc9a4e2ce","Type":"ContainerStarted","Data":"cbb6a9a78ffb36f010515ec91c34539732b19c9aa8277c87e705521bbeb17891"}
Nov 01 22:40:10.679810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:10.679331    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm" event={"ID":"411cba67-a78b-4479-98e8-588fc9a4e2ce","Type":"ContainerStarted","Data":"ccb2438686cfba948cd91917002440591f8140df4114f215384306b53100df00"}
Nov 01 22:40:10.811710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:10.790443    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm" podStartSLOduration=1.790424748 podStartE2EDuration="1.790424748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:10.789260884 +0000 UTC m=+992.106910885" watchObservedRunningTime="2026-11-01 22:40:10.790424748 +0000 UTC m=+992.108074750"
Nov 01 22:40:11.688264 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:11.688232    4531 generic.go:356] "Generic (PLEG): container finished" podID="411cba67-a78b-4479-98e8-588fc9a4e2ce" containerID="cbb6a9a78ffb36f010515ec91c34539732b19c9aa8277c87e705521bbeb17891" exitCode=0
Nov 01 22:40:11.688625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:11.688310    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm" event={"ID":"411cba67-a78b-4479-98e8-588fc9a4e2ce","Type":"ContainerDied","Data":"cbb6a9a78ffb36f010515ec91c34539732b19c9aa8277c87e705521bbeb17891"}
Nov 01 22:40:12.526796 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:40:12.526740    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:40:12.859936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.859912    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:12.932031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.932001    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca\") pod \"411cba67-a78b-4479-98e8-588fc9a4e2ce\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") "
Nov 01 22:40:12.932174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.932051    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv\" (UniqueName: \"kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv\") pod \"411cba67-a78b-4479-98e8-588fc9a4e2ce\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") "
Nov 01 22:40:12.932174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.932115    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca\") pod \"411cba67-a78b-4479-98e8-588fc9a4e2ce\" (UID: \"411cba67-a78b-4479-98e8-588fc9a4e2ce\") "
Nov 01 22:40:12.932400 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.932370    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca" pod "411cba67-a78b-4479-98e8-588fc9a4e2ce" (UID: "411cba67-a78b-4479-98e8-588fc9a4e2ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:12.932510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.932487    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:12.934133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.934113    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca" pod "411cba67-a78b-4479-98e8-588fc9a4e2ce" (UID: "411cba67-a78b-4479-98e8-588fc9a4e2ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:12.934216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:12.934198    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv" pod "411cba67-a78b-4479-98e8-588fc9a4e2ce" (UID: "411cba67-a78b-4479-98e8-588fc9a4e2ce"). InnerVolumeSpecName "kube-api-access-m7wtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:13.034090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:13.034062    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/411cba67-a78b-4479-98e8-588fc9a4e2ce-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:13.034090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:13.034087    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7wtv\" (UniqueName: \"kubernetes.io/projected/411cba67-a78b-4479-98e8-588fc9a4e2ce-kube-api-access-m7wtv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:13.705227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:13.705146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm" event={"ID":"411cba67-a78b-4479-98e8-588fc9a4e2ce","Type":"ContainerDied","Data":"ccb2438686cfba948cd91917002440591f8140df4114f215384306b53100df00"}
Nov 01 22:40:13.705502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:13.705427    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"
Nov 01 22:40:13.705677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:13.705486    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccb2438686cfba948cd91917002440591f8140df4114f215384306b53100df00"
Nov 01 22:40:13.824777 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:13.824739    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:18.658567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:18.658510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:20.085937 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.085899    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"]
Nov 01 22:40:20.086691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.086669    4531 cpu_manager.go:414] "containerMap: removing container" podUID="411cba67-a78b-4479-98e8-588fc9a4e2ce" containerName="revoke-keys"
Nov 01 22:40:20.086817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.086692    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="411cba67-a78b-4479-98e8-588fc9a4e2ce" containerName="revoke-keys"
Nov 01 22:40:20.086883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.086857    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="411cba67-a78b-4479-98e8-588fc9a4e2ce" containerName="revoke-keys"
Nov 01 22:40:20.090711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.090691    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.146022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.145997    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"]
Nov 01 22:40:20.187064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.187037    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:40:20.187188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.187076    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:40:20.206380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.206337    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dq5c\" (UniqueName: \"kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.206488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.206432    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.206530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.206493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.307611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.307581    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.307715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.307621    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.307819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.307803    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dq5c\" (UniqueName: \"kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.308499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.308473    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.309964 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.309947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.361907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.361855    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dq5c\" (UniqueName: \"kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c\") pod \"maas-api-revoke-sub-8bbdc73c-mrll9\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.403948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.403922    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:20.786670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:20.786639    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"]
Nov 01 22:40:20.790902 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:20.790874    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9762f0_db50_4768_ab64_5136ceb3d8df.slice/crio-098e25f3e832bf1841da114df6a6afa842f5d2a5e3ed8a5a3314ff921cbeac2f WatchSource:0}: Error finding container 098e25f3e832bf1841da114df6a6afa842f5d2a5e3ed8a5a3314ff921cbeac2f: Status 404 returned error can't find the container with id 098e25f3e832bf1841da114df6a6afa842f5d2a5e3ed8a5a3314ff921cbeac2f
Nov 01 22:40:21.781813 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:21.781774    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9" event={"ID":"6e9762f0-db50-4768-ab64-5136ceb3d8df","Type":"ContainerStarted","Data":"d07c19dec23cd6624eb6a449259d29e591ff24a4e8eeb2c5c2f947f8f4134af1"}
Nov 01 22:40:21.782159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:21.781891    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9" event={"ID":"6e9762f0-db50-4768-ab64-5136ceb3d8df","Type":"ContainerStarted","Data":"098e25f3e832bf1841da114df6a6afa842f5d2a5e3ed8a5a3314ff921cbeac2f"}
Nov 01 22:40:22.790260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.790223    4531 generic.go:356] "Generic (PLEG): container finished" podID="6e9762f0-db50-4768-ab64-5136ceb3d8df" containerID="d07c19dec23cd6624eb6a449259d29e591ff24a4e8eeb2c5c2f947f8f4134af1" exitCode=0
Nov 01 22:40:22.790716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.790315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9" event={"ID":"6e9762f0-db50-4768-ab64-5136ceb3d8df","Type":"ContainerDied","Data":"d07c19dec23cd6624eb6a449259d29e591ff24a4e8eeb2c5c2f947f8f4134af1"}
Nov 01 22:40:22.937685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.937661    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:22.940949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.940928    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca\") pod \"6e9762f0-db50-4768-ab64-5136ceb3d8df\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") "
Nov 01 22:40:22.941071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.940980    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c\" (UniqueName: \"kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c\") pod \"6e9762f0-db50-4768-ab64-5136ceb3d8df\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") "
Nov 01 22:40:22.941142 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.941085    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca\") pod \"6e9762f0-db50-4768-ab64-5136ceb3d8df\" (UID: \"6e9762f0-db50-4768-ab64-5136ceb3d8df\") "
Nov 01 22:40:22.941447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.941426    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca" pod "6e9762f0-db50-4768-ab64-5136ceb3d8df" (UID: "6e9762f0-db50-4768-ab64-5136ceb3d8df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:22.941613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.941599    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:22.942883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.942857    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca" pod "6e9762f0-db50-4768-ab64-5136ceb3d8df" (UID: "6e9762f0-db50-4768-ab64-5136ceb3d8df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:22.942970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:22.942893    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c" pod "6e9762f0-db50-4768-ab64-5136ceb3d8df" (UID: "6e9762f0-db50-4768-ab64-5136ceb3d8df"). InnerVolumeSpecName "kube-api-access-9dq5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:23.042402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:23.042346    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e9762f0-db50-4768-ab64-5136ceb3d8df-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:23.042402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:23.042369    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dq5c\" (UniqueName: \"kubernetes.io/projected/6e9762f0-db50-4768-ab64-5136ceb3d8df-kube-api-access-9dq5c\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:23.570282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:23.570252    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:23.799368 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:23.799338    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"
Nov 01 22:40:23.799785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:23.799371    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9" event={"ID":"6e9762f0-db50-4768-ab64-5136ceb3d8df","Type":"ContainerDied","Data":"098e25f3e832bf1841da114df6a6afa842f5d2a5e3ed8a5a3314ff921cbeac2f"}
Nov 01 22:40:23.799785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:23.799424    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="098e25f3e832bf1841da114df6a6afa842f5d2a5e3ed8a5a3314ff921cbeac2f"
Nov 01 22:40:24.877970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.877944    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"]
Nov 01 22:40:24.878495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.878481    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6e9762f0-db50-4768-ab64-5136ceb3d8df" containerName="revoke-keys"
Nov 01 22:40:24.878538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.878496    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e9762f0-db50-4768-ab64-5136ceb3d8df" containerName="revoke-keys"
Nov 01 22:40:24.878590 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.878582    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e9762f0-db50-4768-ab64-5136ceb3d8df" containerName="revoke-keys"
Nov 01 22:40:24.881992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.881975    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959180    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959220    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959349    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959405    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959535 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959424    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959567    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959645 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959618    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.959754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.959714    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8f7c\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:24.988039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:24.988011    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"]
Nov 01 22:40:25.060628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060597    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.060817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.060817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060682    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8f7c\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.060817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060756    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.060817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060803    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060838    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060881    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.060971    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.061131    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.061181    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.061265    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.061347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.061797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.061715    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.063251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.063231    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.063465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.063448    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.114553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.114496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.114808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.114789    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8f7c\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.195488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.195468    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:25.627426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.627340    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"]
Nov 01 22:40:25.634540 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:25.634510    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd61e736_df0b_453a_a0c5_afbf11ac90a4.slice/crio-d57a1dcdffbbc0ec1ec7981e28bf5464df3110fc66c4bf7d4ca16215351f3317 WatchSource:0}: Error finding container d57a1dcdffbbc0ec1ec7981e28bf5464df3110fc66c4bf7d4ca16215351f3317: Status 404 returned error can't find the container with id d57a1dcdffbbc0ec1ec7981e28bf5464df3110fc66c4bf7d4ca16215351f3317
Nov 01 22:40:25.636794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.636765    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:40:25.636866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.636825    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:40:25.636866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.636853    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:40:25.818273 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.818241    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" event={"ID":"fd61e736-df0b-453a-a0c5-afbf11ac90a4","Type":"ContainerStarted","Data":"a2f04b780171f59c0212cfb22ac4e57549673b601e5fdd1d5828a06868616710"}
Nov 01 22:40:25.818465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.818297    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" event={"ID":"fd61e736-df0b-453a-a0c5-afbf11ac90a4","Type":"ContainerStarted","Data":"d57a1dcdffbbc0ec1ec7981e28bf5464df3110fc66c4bf7d4ca16215351f3317"}
Nov 01 22:40:25.934789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:25.934683    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podStartSLOduration=1.9346690560000002 podStartE2EDuration="1.934669056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:25.930520556 +0000 UTC m=+1007.248170558" watchObservedRunningTime="2026-11-01 22:40:25.934669056 +0000 UTC m=+1007.252319115"
Nov 01 22:40:26.196100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:26.196072    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:26.197454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:26.197429    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 22:40:26.197573 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:26.197485    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 22:40:27.196603 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:27.196564    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 22:40:27.197044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:27.196638    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 22:40:28.195878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:28.195840    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 22:40:28.196078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:28.195902    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 22:40:28.641618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:28.641538    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:29.196132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:29.196101    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 22:40:29.196316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:29.196156    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 22:40:30.196707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:30.196669    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 22:40:30.197128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:30.196744    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 22:40:31.196209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:31.196171    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body=
Nov 01 22:40:31.196389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:31.196232    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused"
Nov 01 22:40:32.330265 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:32.330232    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:32.331271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:32.331254    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:32.331981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:32.331956    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:40:33.199272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:33.199237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:33.475610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:33.475534    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:35.623458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.623428    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"]
Nov 01 22:40:35.628892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.628876    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.676105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.676081    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.676229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.676110    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.676304 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.676259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz7lf\" (UniqueName: \"kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.687661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.687633    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"]
Nov 01 22:40:35.723546 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.723518    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:40:35.723663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.723575    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:40:35.777435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.777411    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.777435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.777438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.777988 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.777584    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz7lf\" (UniqueName: \"kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.778214 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.778190    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.780186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.780160    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.831642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.831623    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz7lf\" (UniqueName: \"kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf\") pod \"maas-api-revoke-sub-9c3b972d-629zv\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:35.943872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:35.943809    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:36.328383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:36.328357    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"]
Nov 01 22:40:36.331370 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:36.331341    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70dfce34_c0ef_4860_b1cf_a78b9b184374.slice/crio-fa1c4c3f8dc82a9ef51cfd321ff752c62ffa1a5367400b7c3fc4d8a92c669407 WatchSource:0}: Error finding container fa1c4c3f8dc82a9ef51cfd321ff752c62ffa1a5367400b7c3fc4d8a92c669407: Status 404 returned error can't find the container with id fa1c4c3f8dc82a9ef51cfd321ff752c62ffa1a5367400b7c3fc4d8a92c669407
Nov 01 22:40:36.906086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:36.906035    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv" event={"ID":"70dfce34-c0ef-4860-b1cf-a78b9b184374","Type":"ContainerStarted","Data":"555074d756240851cde31f8d7f64738064dd458b61b9b95c8e73f6a1019f3e6d"}
Nov 01 22:40:36.906457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:36.906108    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv" event={"ID":"70dfce34-c0ef-4860-b1cf-a78b9b184374","Type":"ContainerStarted","Data":"fa1c4c3f8dc82a9ef51cfd321ff752c62ffa1a5367400b7c3fc4d8a92c669407"}
Nov 01 22:40:37.015883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:37.015825    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv" podStartSLOduration=2.015809811 podStartE2EDuration="2.015809811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:37.011602989 +0000 UTC m=+1018.329252991" watchObservedRunningTime="2026-11-01 22:40:37.015809811 +0000 UTC m=+1018.333459813"
Nov 01 22:40:37.915088 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:37.915058    4531 generic.go:356] "Generic (PLEG): container finished" podID="70dfce34-c0ef-4860-b1cf-a78b9b184374" containerID="555074d756240851cde31f8d7f64738064dd458b61b9b95c8e73f6a1019f3e6d" exitCode=0
Nov 01 22:40:37.915470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:37.915139    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv" event={"ID":"70dfce34-c0ef-4860-b1cf-a78b9b184374","Type":"ContainerDied","Data":"555074d756240851cde31f8d7f64738064dd458b61b9b95c8e73f6a1019f3e6d"}
Nov 01 22:40:38.299151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:38.299116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:39.117059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.117036    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:39.214759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.214717    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf\" (UniqueName: \"kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf\") pod \"70dfce34-c0ef-4860-b1cf-a78b9b184374\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") "
Nov 01 22:40:39.214883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.214814    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca\") pod \"70dfce34-c0ef-4860-b1cf-a78b9b184374\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") "
Nov 01 22:40:39.214925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.214882    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca\") pod \"70dfce34-c0ef-4860-b1cf-a78b9b184374\" (UID: \"70dfce34-c0ef-4860-b1cf-a78b9b184374\") "
Nov 01 22:40:39.215210 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.215184    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca" pod "70dfce34-c0ef-4860-b1cf-a78b9b184374" (UID: "70dfce34-c0ef-4860-b1cf-a78b9b184374"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:40:39.216812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.216789    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca" pod "70dfce34-c0ef-4860-b1cf-a78b9b184374" (UID: "70dfce34-c0ef-4860-b1cf-a78b9b184374"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:40:39.216916 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.216765    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf" pod "70dfce34-c0ef-4860-b1cf-a78b9b184374" (UID: "70dfce34-c0ef-4860-b1cf-a78b9b184374"). InnerVolumeSpecName "kube-api-access-pz7lf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:40:39.316748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.316704    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz7lf\" (UniqueName: \"kubernetes.io/projected/70dfce34-c0ef-4860-b1cf-a78b9b184374-kube-api-access-pz7lf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:39.316877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.316753    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:39.316877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.316767    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70dfce34-c0ef-4860-b1cf-a78b9b184374-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:40:39.852133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.852105    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"]
Nov 01 22:40:39.852685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.852672    4531 cpu_manager.go:414] "containerMap: removing container" podUID="70dfce34-c0ef-4860-b1cf-a78b9b184374" containerName="revoke-keys"
Nov 01 22:40:39.852771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.852686    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70dfce34-c0ef-4860-b1cf-a78b9b184374" containerName="revoke-keys"
Nov 01 22:40:39.852819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.852778    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="70dfce34-c0ef-4860-b1cf-a78b9b184374" containerName="revoke-keys"
Nov 01 22:40:39.856108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.856087    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.909013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.906872    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-e71d55-openshift-default-dockercfg-696c5\""
Nov 01 22:40:39.922577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.922688 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922591    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.922688 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922665    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.922800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922711    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.922836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922806    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.922875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922833    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.922989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.922973    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcn4g\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.923024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.923012    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.923085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.923029    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:39.935209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.935192    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"
Nov 01 22:40:39.935330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.935235    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv" event={"ID":"70dfce34-c0ef-4860-b1cf-a78b9b184374","Type":"ContainerDied","Data":"fa1c4c3f8dc82a9ef51cfd321ff752c62ffa1a5367400b7c3fc4d8a92c669407"}
Nov 01 22:40:39.935330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.935314    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa1c4c3f8dc82a9ef51cfd321ff752c62ffa1a5367400b7c3fc4d8a92c669407"
Nov 01 22:40:39.959057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:39.959028    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"]
Nov 01 22:40:40.024136 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024113    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcn4g\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024151    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024169    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024207    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024227    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024248    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024269    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024306    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024342    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024798    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024882    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024946    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.024999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.024965    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.025089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.025031    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.026702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.026674    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.026847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.026830    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.078338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.078309    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.078466 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.078449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcn4g\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.171759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.171686    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:40.613957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.613932    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"]
Nov 01 22:40:40.616117 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:40.616083    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f1f5981_2381_4f38_9fcb_38fd848a8767.slice/crio-c996b0c28e2de9ac16ba0cdcdabb9f918ab801fe90bf5bc68d20585765de85a0 WatchSource:0}: Error finding container c996b0c28e2de9ac16ba0cdcdabb9f918ab801fe90bf5bc68d20585765de85a0: Status 404 returned error can't find the container with id c996b0c28e2de9ac16ba0cdcdabb9f918ab801fe90bf5bc68d20585765de85a0
Nov 01 22:40:40.619020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.618991    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:40:40.619100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.619061    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:40:40.619100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.619088    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:40:40.945030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.944998    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" event={"ID":"2f1f5981-2381-4f38-9fcb-38fd848a8767","Type":"ContainerStarted","Data":"3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d"}
Nov 01 22:40:40.945198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:40.945053    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" event={"ID":"2f1f5981-2381-4f38-9fcb-38fd848a8767","Type":"ContainerStarted","Data":"c996b0c28e2de9ac16ba0cdcdabb9f918ab801fe90bf5bc68d20585765de85a0"}
Nov 01 22:40:41.058660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:41.058607    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podStartSLOduration=2.058590451 podStartE2EDuration="2.058590451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:41.054896552 +0000 UTC m=+1022.372546566" watchObservedRunningTime="2026-11-01 22:40:41.058590451 +0000 UTC m=+1022.376240453"
Nov 01 22:40:41.172012 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:41.171978    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:41.173824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:41.173794    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:41.173987 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:41.173840    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:42.172556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:42.172520    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:42.173196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:42.173162    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:42.530209 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:40:42.530173    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:40:43.179159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:43.179067    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:43.179716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:43.179682    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:43.266046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:43.266007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:44.181751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:44.178034    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:44.181751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:44.178090    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:45.178753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:45.176005    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:45.178753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:45.176073    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:46.179073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:46.179028    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:46.179642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:46.179611    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:47.172107 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:47.172068    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body=
Nov 01 22:40:47.172283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:47.172144    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused"
Nov 01 22:40:48.132783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:48.132747    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:48.177419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:48.177387    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:48.177915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:48.177890    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:48.178678 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:48.178656    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:40:50.476626 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.476549    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"]
Nov 01 22:40:50.481194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.481178    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.537222 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.537188    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"]
Nov 01 22:40:50.576384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.576357    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"openshift-service-ca.crt\""
Nov 01 22:40:50.576534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.576386    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"kube-root-ca.crt\""
Nov 01 22:40:50.576534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.576436    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"distinct-w6-a6dd2f-kserve-self-signed-certs\""
Nov 01 22:40:50.576534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.576444    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-a6dd2f\"/\"default-dockercfg-gptqw\""
Nov 01 22:40:50.625090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.625042    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:50.660240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.660208    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.660410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.660317    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.660410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.660344    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.660410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.660372    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.660410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.660389    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.660569 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.660497    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5snw\" (UniqueName: \"kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.761515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.761427    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5snw\" (UniqueName: \"kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.761679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.761552    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.761679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.761639    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.761679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.761664    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.761872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.761790    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.761872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.761840    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.762136 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.762111    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.762278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.762187    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.762278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.762228    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.764139 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.764117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.764226 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.764139    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:50.815374 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:50.815336    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5snw\" (UniqueName: \"kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw\") pod \"distinct-w6-a6dd2f-kserve-54f69f56bd-w848h\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") " pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:51.094266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:51.094192    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:51.479311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:51.479283    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"]
Nov 01 22:40:51.482936 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:51.482909    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75ef3a0f_3447_4ed5_85d0_b36c86bbe5db.slice/crio-6f8dbfcb1774669b96592fccf5151350d641a6f31d4b90575eea13b959e91560 WatchSource:0}: Error finding container 6f8dbfcb1774669b96592fccf5151350d641a6f31d4b90575eea13b959e91560: Status 404 returned error can't find the container with id 6f8dbfcb1774669b96592fccf5151350d641a6f31d4b90575eea13b959e91560
Nov 01 22:40:52.049744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:52.049701    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" event={"ID":"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db","Type":"ContainerStarted","Data":"215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0"}
Nov 01 22:40:52.049921 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:52.049763    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" event={"ID":"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db","Type":"ContainerStarted","Data":"6f8dbfcb1774669b96592fccf5151350d641a6f31d4b90575eea13b959e91560"}
Nov 01 22:40:52.094584 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:52.094554    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:40:52.159172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:52.159124    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podStartSLOduration=2.159112883 podStartE2EDuration="2.159112883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:40:52.158143897 +0000 UTC m=+1033.475793899" watchObservedRunningTime="2026-11-01 22:40:52.159112883 +0000 UTC m=+1033.476762884"
Nov 01 22:40:52.958767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:52.958716    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:54.146110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:54.146074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:57.932457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:57.932418    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:58.709046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.709014    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"]
Nov 01 22:40:58.713374 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.713354    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.741196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.741168    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.741342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.741206    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.741342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.741241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.741471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.741363    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.741471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.741421    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.741578 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.741554    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czcgv\" (UniqueName: \"kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.772472 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.772444    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"]
Nov 01 22:40:58.814332 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.814294    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c768f3\"/\"distinct-w2-c768f3-kserve-self-signed-certs\""
Nov 01 22:40:58.842196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.842161    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.842365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.842243    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czcgv\" (UniqueName: \"kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.842594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.842560    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.842762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.842601    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.842762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.842646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.842762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.842741    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.843221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.843197    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.843346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.843237    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.843346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.843279    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.846109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.846069    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.846479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.846459    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:58.877926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.877892    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:40:58.902489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:58.902456    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czcgv\" (UniqueName: \"kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv\") pod \"distinct-w2-c768f3-kserve-978cd7c68-85qqd\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") " pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:59.028272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:59.028178    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:40:59.487200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:40:59.487175    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"]
Nov 01 22:40:59.489606 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:40:59.489575    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47f40cac_b1dc_407e_baed_3256eb2f0699.slice/crio-61f9dbd9c7616ee688cc7c01a682da14a151da3e09013a54e7b58203513c7742 WatchSource:0}: Error finding container 61f9dbd9c7616ee688cc7c01a682da14a151da3e09013a54e7b58203513c7742: Status 404 returned error can't find the container with id 61f9dbd9c7616ee688cc7c01a682da14a151da3e09013a54e7b58203513c7742
Nov 01 22:41:00.134948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:00.134914    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" event={"ID":"47f40cac-b1dc-407e-baed-3256eb2f0699","Type":"ContainerStarted","Data":"a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b"}
Nov 01 22:41:00.135329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:00.134963    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" event={"ID":"47f40cac-b1dc-407e-baed-3256eb2f0699","Type":"ContainerStarted","Data":"61f9dbd9c7616ee688cc7c01a682da14a151da3e09013a54e7b58203513c7742"}
Nov 01 22:41:00.248189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:00.248135    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podStartSLOduration=2.248120598 podStartE2EDuration="2.248120598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:00.24714037 +0000 UTC m=+1041.564790371" watchObservedRunningTime="2026-11-01 22:41:00.248120598 +0000 UTC m=+1041.565770599"
Nov 01 22:41:01.028975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:01.028944    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:41:01.094560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:01.094532    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:41:01.108248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:01.108224    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:41:01.158166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:01.158136    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:41:02.868715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:02.868681    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:06.670885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:06.670855    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:07.745587 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:07.745550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:09.028595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:09.028565    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:41:09.041070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:09.041048    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:41:09.222188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:09.222157    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:41:11.827500 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:11.827467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:12.532880 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:41:12.532839    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:41:12.741319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:12.741277    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:17.647452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:17.647419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:22.501002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:22.500961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:24.973512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:24.973474    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"]
Nov 01 22:41:24.979927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:24.979903    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.004766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.004738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.004878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.004808    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.004878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.004841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.004965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.004880    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.005003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.004981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.005052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.005007    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c57vj\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.005142 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.005123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.005203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.005156    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.005244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.005214    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.033366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.033343    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-a6dd2f-openshift-default-dockercfg-vvpcs\""
Nov 01 22:41:25.083781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.083759    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"]
Nov 01 22:41:25.105828 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.105809    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.105924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.105833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.105924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.105860    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.105924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.105896    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.105956    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.105996    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106059    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106099    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106126    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c57vj\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106220    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106550    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106652 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106561    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.106793 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.106772    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.108124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.108108    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.108619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.108597    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.172501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.172469    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.182884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.182857    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c57vj\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.294879 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.294815    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:25.737570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.737538    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"]
Nov 01 22:41:25.738833 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:41:25.738803    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f006c60_0006_4d13_a623_2d4cb888b717.slice/crio-3176ea56753009dfdeeb75837860451cd35203d7693023a367d67ae3dd0ed143 WatchSource:0}: Error finding container 3176ea56753009dfdeeb75837860451cd35203d7693023a367d67ae3dd0ed143: Status 404 returned error can't find the container with id 3176ea56753009dfdeeb75837860451cd35203d7693023a367d67ae3dd0ed143
Nov 01 22:41:25.741276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.741240    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:25.741392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.741309    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:25.741392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:25.741340    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:26.403937 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:26.403842    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" event={"ID":"0f006c60-0006-4d13-a623-2d4cb888b717","Type":"ContainerStarted","Data":"ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59"}
Nov 01 22:41:26.403937 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:26.403920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" event={"ID":"0f006c60-0006-4d13-a623-2d4cb888b717","Type":"ContainerStarted","Data":"3176ea56753009dfdeeb75837860451cd35203d7693023a367d67ae3dd0ed143"}
Nov 01 22:41:27.296951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:27.296846    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:27.298801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:27.298505    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:41:27.298801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:27.298577    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 22:41:27.509815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:27.504518    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:28.296180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:28.296033    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:41:28.296180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:28.296098    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 22:41:28.556356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:28.556270    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:29.295780 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:29.295362    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:41:29.295780 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:29.295418    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 22:41:30.295432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:30.295394    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:41:30.295851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:30.295461    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 22:41:31.297130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:31.297089    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:41:31.297557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:31.297151    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 22:41:32.295863 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:32.295827    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused" start-of-body=
Nov 01 22:41:32.296066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:32.295877    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.99:15021/healthz/ready\": dial tcp 10.133.0.99:15021: connect: connection refused"
Nov 01 22:41:32.461178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:32.461139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:33.300252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:33.300223    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:33.300586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:33.300565    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:33.301153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:33.301133    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:41:33.412684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:33.412638    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podStartSLOduration=9.412626788 podStartE2EDuration="9.412626788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:26.610677385 +0000 UTC m=+1067.928327387" watchObservedRunningTime="2026-11-01 22:41:33.412626788 +0000 UTC m=+1074.730276789"
Nov 01 22:41:33.519071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:33.519038    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:37.231052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:37.231012    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:42.029812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:42.029773    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:42.528953 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:41:42.528916    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:41:43.153002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.152926    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"]
Nov 01 22:41:43.158766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.158713    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.207837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.207809    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c768f3-openshift-default-dockercfg-r664d\""
Nov 01 22:41:43.260090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.260064    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"]
Nov 01 22:41:43.333390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333339    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333441    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333486    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333520    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333574    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4szzv\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333677    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333718    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.333819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.333767    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434526    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434562    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434668    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434687    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434711    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434774    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434797    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.434910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.434833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4szzv\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.435272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.435122    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.435272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.435246    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.435387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.435301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.435449 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.435399    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.436328 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.436301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.437327 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.437309    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.437607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.437588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.463124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.463098    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"]
Nov 01 22:41:43.468585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.468569    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.501252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.501228    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.501860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.501841    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4szzv\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.524057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.524041    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9dfe9f-openshift-default-dockercfg-9r4dj\""
Nov 01 22:41:43.536553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.536527    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"]
Nov 01 22:41:43.637586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637606    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637826    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637916    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7k7h8\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637953    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.637991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.637981    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.638073 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.638039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.738956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.738933    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.738960    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739062    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739130    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739147    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739174    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739213    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7k7h8\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739490 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739263    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739490 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739377    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739487    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739569    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739677    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.739888 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.739868    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.741396 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.741372    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.741606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.741588    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.772901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.772873    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:43.793551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.793529    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7k7h8\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:43.793907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:43.793885    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:44.083286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.083213    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:44.209902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.209868    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"]
Nov 01 22:41:44.213316 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:41:44.213277    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba4fbea_c951_4772_849f_80f6eb700609.slice/crio-6e2a83078df8d483801346ef72205a713cc4e252b124cef5c13979f9c0479a32 WatchSource:0}: Error finding container 6e2a83078df8d483801346ef72205a713cc4e252b124cef5c13979f9c0479a32: Status 404 returned error can't find the container with id 6e2a83078df8d483801346ef72205a713cc4e252b124cef5c13979f9c0479a32
Nov 01 22:41:44.216480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.216442    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:44.216563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.216533    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:44.216612 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.216580    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:44.479487 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.479457    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"]
Nov 01 22:41:44.484707 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:41:44.484679    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16b3f34_867c_4036_ac6e_7151754515ad.slice/crio-808ca96d5d413430a8d31302e580232879f367831ed69a79f33233d657121394 WatchSource:0}: Error finding container 808ca96d5d413430a8d31302e580232879f367831ed69a79f33233d657121394: Status 404 returned error can't find the container with id 808ca96d5d413430a8d31302e580232879f367831ed69a79f33233d657121394
Nov 01 22:41:44.487897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.487861    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:44.487989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.487943    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:44.488029 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.487992    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:44.567526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.567501    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" event={"ID":"d16b3f34-867c-4036-ac6e-7151754515ad","Type":"ContainerStarted","Data":"808ca96d5d413430a8d31302e580232879f367831ed69a79f33233d657121394"}
Nov 01 22:41:44.569377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.569355    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" event={"ID":"8ba4fbea-c951-4772-849f-80f6eb700609","Type":"ContainerStarted","Data":"c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405"}
Nov 01 22:41:44.569484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.569401    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" event={"ID":"8ba4fbea-c951-4772-849f-80f6eb700609","Type":"ContainerStarted","Data":"6e2a83078df8d483801346ef72205a713cc4e252b124cef5c13979f9c0479a32"}
Nov 01 22:41:44.683404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.683335    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podStartSLOduration=1.683316339 podStartE2EDuration="1.683316339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:44.680387208 +0000 UTC m=+1085.998037211" watchObservedRunningTime="2026-11-01 22:41:44.683316339 +0000 UTC m=+1086.000966341"
Nov 01 22:41:44.773680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:44.773592    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:45.627826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:45.627108    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" event={"ID":"d16b3f34-867c-4036-ac6e-7151754515ad","Type":"ContainerStarted","Data":"fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc"}
Nov 01 22:41:45.788833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:45.788799    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:41:45.789106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:45.789075    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 22:41:45.839787 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:45.839705    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podStartSLOduration=2.839684911 podStartE2EDuration="2.839684911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:45.788690818 +0000 UTC m=+1087.106340825" watchObservedRunningTime="2026-11-01 22:41:45.839684911 +0000 UTC m=+1087.157334909"
Nov 01 22:41:46.087245 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:46.087211    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:46.087649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:46.087624    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 22:41:46.087877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:46.087852    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 22:41:46.784751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:46.784082    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:41:46.784751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:46.784139    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 22:41:47.089808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:47.089533    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 22:41:47.089808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:47.089620    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 22:41:47.112748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:47.112591    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:47.775148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:47.775109    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:41:47.775423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:47.775396    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 22:41:48.092803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:48.092095    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 22:41:48.092803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:48.092157    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 22:41:48.786760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:48.786103    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:41:48.786760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:48.786165    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 22:41:49.089960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:49.088055    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 22:41:49.089960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:49.088118    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 22:41:49.776759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:49.776080    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:41:49.776759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:49.776144    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 22:41:50.085915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:50.085085    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 22:41:50.085915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:50.085144    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 22:41:50.776751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:50.775139    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Nov 01 22:41:50.776751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:50.775205    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Nov 01 22:41:51.084624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:51.084537    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body=
Nov 01 22:41:51.084624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:51.084592    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused"
Nov 01 22:41:52.234194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.233708    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:52.234711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.234537    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:52.235132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.235112    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:41:52.374908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.374866    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:52.429235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.429192    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:52.431833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.430334    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:52.433146 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:52.433124    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:41:55.116770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.116737    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"]
Nov 01 22:41:55.121176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.121143    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294430    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4v6n\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294469    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294574    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294640    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294716    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294765    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294810    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294841    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.294956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.294865    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.360748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.357595    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"]
Nov 01 22:41:55.396650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396552    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4v6n\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396599    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396649    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396691    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396783    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396804    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396832    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396870    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.396905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.396903    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.398185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.397491    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.398185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.398124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.398185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.398134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.398895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.398861    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.399990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.399972    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.402352 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.402319    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.402634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.402607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.451527 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.451497    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.451817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.451793    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4v6n\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:55.737234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:55.737206    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:56.174607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.174445    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"]
Nov 01 22:41:56.177825 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:41:56.177794    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfef54cec_28ed_4f26_b498_27a0012e2f5c.slice/crio-dfec9d326a50c950f07abf6483dafdd8de09c09139f639d3345ed85097c64d11 WatchSource:0}: Error finding container dfec9d326a50c950f07abf6483dafdd8de09c09139f639d3345ed85097c64d11: Status 404 returned error can't find the container with id dfec9d326a50c950f07abf6483dafdd8de09c09139f639d3345ed85097c64d11
Nov 01 22:41:56.180791 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.180759    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:56.180889 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.180822    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:56.180889 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.180851    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:41:56.750624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.750590    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" event={"ID":"fef54cec-28ed-4f26-b498-27a0012e2f5c","Type":"ContainerStarted","Data":"d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1"}
Nov 01 22:41:56.750844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.750644    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" event={"ID":"fef54cec-28ed-4f26-b498-27a0012e2f5c","Type":"ContainerStarted","Data":"dfec9d326a50c950f07abf6483dafdd8de09c09139f639d3345ed85097c64d11"}
Nov 01 22:41:56.912851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:56.912799    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podStartSLOduration=1.91278448 podStartE2EDuration="1.91278448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:41:56.908810863 +0000 UTC m=+1098.226460856" watchObservedRunningTime="2026-11-01 22:41:56.91278448 +0000 UTC m=+1098.230434477"
Nov 01 22:41:57.127823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:57.127701    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:41:57.737419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:57.737376    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:41:57.738812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:57.738784    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:41:57.738961 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:57.738831    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 22:41:58.738464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:58.738423    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:41:58.738971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:58.738482    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 22:41:59.737921 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:59.737889    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:41:59.738090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:41:59.737939    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 22:42:00.737650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:00.737620    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:42:00.738054 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:00.737671    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 22:42:00.825311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:00.825255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:01.738251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:01.738218    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:42:01.738618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:01.738291    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 22:42:02.186809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:02.186106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:02.737809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:02.737769    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body=
Nov 01 22:42:02.737971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:02.737835    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused"
Nov 01 22:42:03.742240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:03.742209    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:42:03.742742 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:03.742597    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:42:03.743347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:03.743321    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:42:07.060750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:07.060646    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:12.082004 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:12.081921    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:12.528241 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:42:12.528208    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:42:14.062683 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:14.062649    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"]
Nov 01 22:42:14.109103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:14.109070    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e0d63fe-4r2v7"]
Nov 01 22:42:14.820231 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:14.820201    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0201c0d-aac6-4d65-b628-8d43107ce507" path="/var/lib/kubelet/pods/e0201c0d-aac6-4d65-b628-8d43107ce507/volumes"
Nov 01 22:42:16.837653 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:16.837616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:21.639365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:21.639268    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:26.482023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:26.481935    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:31.274837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:31.274799    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:33.652991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:33.652952    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:33.886645 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:33.886608    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:36.041659 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:36.041591    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:40.049974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:40.049922    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"]
Nov 01 22:42:40.095541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:40.095502    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-073ddf60-b6zn4"]
Nov 01 22:42:40.834319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:40.834254    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:40.841541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:40.841456    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2764ac6-d90e-41bf-b68d-936925bf8d55" path="/var/lib/kubelet/pods/b2764ac6-d90e-41bf-b68d-936925bf8d55/volumes"
Nov 01 22:42:42.532096 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:42:42.532056    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:42:42.758583 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.758553    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"]
Nov 01 22:42:42.764027 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.764008    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.825716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.825638    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"]
Nov 01 22:42:42.859973 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.859923    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:42:42.860110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.859982    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:42:42.871317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.871279    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.871445 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.871325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p5q2\" (UniqueName: \"kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.871595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.871568    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.972539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.972506    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.972799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.972777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.972917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.972810    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p5q2\" (UniqueName: \"kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.973551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.973528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:42.975024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:42.974993    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:43.028270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:43.028243    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p5q2\" (UniqueName: \"kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2\") pod \"maas-api-revoke-sub-05d3d22d-tx9fv\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:43.078103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:43.078047    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:43.466644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:43.466611    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"]
Nov 01 22:42:43.468807 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:42:43.468778    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251c836c_f80a_497a_ae5e_a381c12bf8d9.slice/crio-a7f7bc1a7261d37022970c763de386d604fe0bd88b3b757874e8bf5e4232f98e WatchSource:0}: Error finding container a7f7bc1a7261d37022970c763de386d604fe0bd88b3b757874e8bf5e4232f98e: Status 404 returned error can't find the container with id a7f7bc1a7261d37022970c763de386d604fe0bd88b3b757874e8bf5e4232f98e
Nov 01 22:42:44.146811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:44.146721    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv" event={"ID":"251c836c-f80a-497a-ae5e-a381c12bf8d9","Type":"ContainerStarted","Data":"eddee2ebfe5a26d8be0205fb296ffbfa69cadf3be8fc6538069be65583405a53"}
Nov 01 22:42:44.146811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:44.146797    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv" event={"ID":"251c836c-f80a-497a-ae5e-a381c12bf8d9","Type":"ContainerStarted","Data":"a7f7bc1a7261d37022970c763de386d604fe0bd88b3b757874e8bf5e4232f98e"}
Nov 01 22:42:44.259523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:44.259466    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv" podStartSLOduration=2.259452745 podStartE2EDuration="2.259452745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:44.257158626 +0000 UTC m=+1145.574808628" watchObservedRunningTime="2026-11-01 22:42:44.259452745 +0000 UTC m=+1145.577102745"
Nov 01 22:42:45.157476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:45.157440    4531 generic.go:356] "Generic (PLEG): container finished" podID="251c836c-f80a-497a-ae5e-a381c12bf8d9" containerID="eddee2ebfe5a26d8be0205fb296ffbfa69cadf3be8fc6538069be65583405a53" exitCode=0
Nov 01 22:42:45.157919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:45.157521    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv" event={"ID":"251c836c-f80a-497a-ae5e-a381c12bf8d9","Type":"ContainerDied","Data":"eddee2ebfe5a26d8be0205fb296ffbfa69cadf3be8fc6538069be65583405a53"}
Nov 01 22:42:45.573066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:45.573026    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:46.347057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.347032    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:46.518208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.518176    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca\") pod \"251c836c-f80a-497a-ae5e-a381c12bf8d9\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") "
Nov 01 22:42:46.518388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.518268    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca\") pod \"251c836c-f80a-497a-ae5e-a381c12bf8d9\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") "
Nov 01 22:42:46.518388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.518327    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2\" (UniqueName: \"kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2\") pod \"251c836c-f80a-497a-ae5e-a381c12bf8d9\" (UID: \"251c836c-f80a-497a-ae5e-a381c12bf8d9\") "
Nov 01 22:42:46.518528 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.518505    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca" pod "251c836c-f80a-497a-ae5e-a381c12bf8d9" (UID: "251c836c-f80a-497a-ae5e-a381c12bf8d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:42:46.518662 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.518646    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:42:46.521208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.521180    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2" pod "251c836c-f80a-497a-ae5e-a381c12bf8d9" (UID: "251c836c-f80a-497a-ae5e-a381c12bf8d9"). InnerVolumeSpecName "kube-api-access-8p5q2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:42:46.521208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.521194    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca" pod "251c836c-f80a-497a-ae5e-a381c12bf8d9" (UID: "251c836c-f80a-497a-ae5e-a381c12bf8d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:42:46.619544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.619522    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p5q2\" (UniqueName: \"kubernetes.io/projected/251c836c-f80a-497a-ae5e-a381c12bf8d9-kube-api-access-8p5q2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:42:46.619544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:46.619545    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c836c-f80a-497a-ae5e-a381c12bf8d9-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:42:47.178363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:47.178284    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"
Nov 01 22:42:47.178363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:47.178288    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv" event={"ID":"251c836c-f80a-497a-ae5e-a381c12bf8d9","Type":"ContainerDied","Data":"a7f7bc1a7261d37022970c763de386d604fe0bd88b3b757874e8bf5e4232f98e"}
Nov 01 22:42:47.178363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:47.178348    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7f7bc1a7261d37022970c763de386d604fe0bd88b3b757874e8bf5e4232f98e"
Nov 01 22:42:50.481608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:50.481569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:55.056444 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:55.056410    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"]
Nov 01 22:42:55.102865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:55.102833    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2523a479-h56v5"]
Nov 01 22:42:55.253921 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:55.253884    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:42:56.816946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:56.816910    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a" path="/var/lib/kubelet/pods/a8bf1ce1-16e2-4b28-89e1-6ae37bd9350a/volumes"
Nov 01 22:42:57.696650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.696618    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"]
Nov 01 22:42:57.697247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.697233    4531 cpu_manager.go:414] "containerMap: removing container" podUID="251c836c-f80a-497a-ae5e-a381c12bf8d9" containerName="revoke-keys"
Nov 01 22:42:57.697306 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.697249    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251c836c-f80a-497a-ae5e-a381c12bf8d9" containerName="revoke-keys"
Nov 01 22:42:57.697361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.697352    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="251c836c-f80a-497a-ae5e-a381c12bf8d9" containerName="revoke-keys"
Nov 01 22:42:57.701173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.701157    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.751504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.751475    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.751633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.751603    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.751798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.751780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9rbp\" (UniqueName: \"kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.797577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.797551    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:42:57.797694 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.797555    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:42:57.802078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.802056    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"]
Nov 01 22:42:57.852788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.852756    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.853195 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.852980    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9rbp\" (UniqueName: \"kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.853281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.853199    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.853479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.853458    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.855390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.855368    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:57.906754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:57.906704    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9rbp\" (UniqueName: \"kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp\") pod \"maas-api-revoke-sub-41f39c92-zckkp\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:58.015202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:58.015149    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:42:58.400323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:58.400291    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"]
Nov 01 22:42:58.402203 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:42:58.402174    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec3f78e_c1f2_42ec_81e6_21ba7829f0a7.slice/crio-3d8819659968934e0675ed7d51e8d7e74186dc1440c9c74f7af796048426ee84 WatchSource:0}: Error finding container 3d8819659968934e0675ed7d51e8d7e74186dc1440c9c74f7af796048426ee84: Status 404 returned error can't find the container with id 3d8819659968934e0675ed7d51e8d7e74186dc1440c9c74f7af796048426ee84
Nov 01 22:42:59.294441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:59.294367    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp" event={"ID":"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7","Type":"ContainerStarted","Data":"703d4c184040b2dec15fb84b605a2354e5139962ab52d8f24c1368d4d6a7ac88"}
Nov 01 22:42:59.294846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:59.294549    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp" event={"ID":"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7","Type":"ContainerStarted","Data":"3d8819659968934e0675ed7d51e8d7e74186dc1440c9c74f7af796048426ee84"}
Nov 01 22:42:59.405044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:42:59.404983    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp" podStartSLOduration=2.4049667 podStartE2EDuration="2.4049667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:42:59.403981911 +0000 UTC m=+1160.721631913" watchObservedRunningTime="2026-11-01 22:42:59.4049667 +0000 UTC m=+1160.722616702"
Nov 01 22:43:00.058614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:00.058532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:00.303329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:00.303299    4531 generic.go:356] "Generic (PLEG): container finished" podID="eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" containerID="703d4c184040b2dec15fb84b605a2354e5139962ab52d8f24c1368d4d6a7ac88" exitCode=0
Nov 01 22:43:00.303747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:00.303380    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp" event={"ID":"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7","Type":"ContainerDied","Data":"703d4c184040b2dec15fb84b605a2354e5139962ab52d8f24c1368d4d6a7ac88"}
Nov 01 22:43:01.463989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.463967    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:43:01.599281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.599148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca\") pod \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") "
Nov 01 22:43:01.599281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.599263    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca\") pod \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") "
Nov 01 22:43:01.599523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.599283    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp\" (UniqueName: \"kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp\") pod \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\" (UID: \"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7\") "
Nov 01 22:43:01.599914 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.599879    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca" pod "eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" (UID: "eec3f78e-c1f2-42ec-81e6-21ba7829f0a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:43:01.601217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.601179    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca" pod "eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" (UID: "eec3f78e-c1f2-42ec-81e6-21ba7829f0a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:43:01.601464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.601440    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp" pod "eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" (UID: "eec3f78e-c1f2-42ec-81e6-21ba7829f0a7"). InnerVolumeSpecName "kube-api-access-r9rbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:43:01.701438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.701409    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:01.701438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.701437    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:01.701628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:01.701447    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9rbp\" (UniqueName: \"kubernetes.io/projected/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7-kube-api-access-r9rbp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:02.321525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:02.321493    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"
Nov 01 22:43:02.321696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:02.321524    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp" event={"ID":"eec3f78e-c1f2-42ec-81e6-21ba7829f0a7","Type":"ContainerDied","Data":"3d8819659968934e0675ed7d51e8d7e74186dc1440c9c74f7af796048426ee84"}
Nov 01 22:43:02.321696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:02.321572    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d8819659968934e0675ed7d51e8d7e74186dc1440c9c74f7af796048426ee84"
Nov 01 22:43:04.954844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:04.954809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:09.868712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:09.868675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:10.103443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:10.103407    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"]
Nov 01 22:43:10.147188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:10.147108    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e2a3bb4-cxtp4"]
Nov 01 22:43:10.818082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:10.818040    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec944f0f-cc3a-4a19-b469-98a8dd10b0d2" path="/var/lib/kubelet/pods/ec944f0f-cc3a-4a19-b469-98a8dd10b0d2/volumes"
Nov 01 22:43:12.529703 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:43:12.529669    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:43:14.858943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:14.858905    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:15.835569 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.835539    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"]
Nov 01 22:43:15.836162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.836149    4531 cpu_manager.go:414] "containerMap: removing container" podUID="eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" containerName="revoke-keys"
Nov 01 22:43:15.836220 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.836165    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" containerName="revoke-keys"
Nov 01 22:43:15.836262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.836253    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" containerName="revoke-keys"
Nov 01 22:43:15.840967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.840947    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:15.936851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.936819    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9dfe9f\"/\"distinct-2-w4-9dfe9f-kserve-self-signed-certs\""
Nov 01 22:43:15.941266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.941242    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"]
Nov 01 22:43:15.961657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.961632    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m4gx\" (UniqueName: \"kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:15.961788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.961688    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:15.961835 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.961797    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:15.961835 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.961827    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:15.961916 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.961852    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:15.961957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:15.961934    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.003006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.002962    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:16.063681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.063648    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m4gx\" (UniqueName: \"kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.063890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.063712    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.063890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.063750    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.063890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.063767    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.063890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.063792    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.063890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.063856    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.064259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.064225    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.064259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.064245    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.064398 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.064331    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.066271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.066250    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.066394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.066375    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.127754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.127663    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m4gx\" (UniqueName: \"kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx\") pod \"distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") " pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.154146 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.154117    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:16.589326 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:16.589296    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"]
Nov 01 22:43:16.593652 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:43:16.593623    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00109ff1_30af_470b_a0b1_a1b47d19ac07.slice/crio-878378f1c8ba9663d62281acfc53dae7abeee7ae8720d87fa4dc059a3f1250a2 WatchSource:0}: Error finding container 878378f1c8ba9663d62281acfc53dae7abeee7ae8720d87fa4dc059a3f1250a2: Status 404 returned error can't find the container with id 878378f1c8ba9663d62281acfc53dae7abeee7ae8720d87fa4dc059a3f1250a2
Nov 01 22:43:17.448833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:17.448796    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" event={"ID":"00109ff1-30af-470b-a0b1-a1b47d19ac07","Type":"ContainerStarted","Data":"2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164"}
Nov 01 22:43:17.449216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:17.448839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" event={"ID":"00109ff1-30af-470b-a0b1-a1b47d19ac07","Type":"ContainerStarted","Data":"878378f1c8ba9663d62281acfc53dae7abeee7ae8720d87fa4dc059a3f1250a2"}
Nov 01 22:43:17.561860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:17.561805    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podStartSLOduration=2.56179126 podStartE2EDuration="2.56179126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:17.559844839 +0000 UTC m=+1178.877494852" watchObservedRunningTime="2026-11-01 22:43:17.56179126 +0000 UTC m=+1178.879441261"
Nov 01 22:43:18.154817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:18.154778    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:19.835186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:19.834594    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:22.061349 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:22.061290    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:22.677313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:22.677271    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:24.629663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:24.629626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:25.851858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.851825    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"]
Nov 01 22:43:25.856027 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.856005    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:25.896777 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.896747    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"]
Nov 01 22:43:25.900803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.900784    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.913502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.913478    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:43:25.915693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.915669    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"]
Nov 01 22:43:25.938361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.938344    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:43:25.968712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.968686    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"]
Nov 01 22:43:25.993658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.993623    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.993815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.993712    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.993815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.993750    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.993815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.993798    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2x6s\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.994001 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.993860    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6xbs\" (UniqueName: \"kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:25.994001 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.993983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.994118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.994025    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:25.994175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.994119    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.994175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.994159    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.994287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.994191    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:25.994287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.994242    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:25.994287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:25.994268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.009951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.009933    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-d6101b-openshift-default-dockercfg-jbnc7\""
Nov 01 22:43:26.095814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.095792    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.095925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.095827    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.095925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.095851    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.095925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.095898    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.095925    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096029    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096081    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096178    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2x6s\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096206    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6xbs\" (UniqueName: \"kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.096255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096243    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096262    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096271    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.096402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096335    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096631 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096607    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.096797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096774    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.096942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.096920    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.097135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.097113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.098480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.098459    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.098785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.098764    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.098895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.098880    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.150098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.150041    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.150353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.150333    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6xbs\" (UniqueName: \"kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs\") pod \"maas-api-revoke-sub-0ab49442-9q8hs\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.151850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.151829    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2x6s\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.154627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.154602    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:26.167692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.167675    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:26.175053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.175037    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:26.216235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.216208    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:26.538032 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.538003    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:43:26.557865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.557838    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"]
Nov 01 22:43:26.561577 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:43:26.561550    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f14e2f_e098_4849_bbb6_d8f4e75931bd.slice/crio-f2d7b35dc4845bf419428ec5b271a2acdf7d14f9e95fdc36bc991c6491f3fd3b WatchSource:0}: Error finding container f2d7b35dc4845bf419428ec5b271a2acdf7d14f9e95fdc36bc991c6491f3fd3b: Status 404 returned error can't find the container with id f2d7b35dc4845bf419428ec5b271a2acdf7d14f9e95fdc36bc991c6491f3fd3b
Nov 01 22:43:26.635005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.634981    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"]
Nov 01 22:43:26.639130 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:43:26.639104    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3639a8b0_f32f_464b_8917_a3d5656f56a2.slice/crio-96ae85161d933ed1094039b891721734d71665733dcda1cdd54a7bbe51f11318 WatchSource:0}: Error finding container 96ae85161d933ed1094039b891721734d71665733dcda1cdd54a7bbe51f11318: Status 404 returned error can't find the container with id 96ae85161d933ed1094039b891721734d71665733dcda1cdd54a7bbe51f11318
Nov 01 22:43:26.641767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.641720    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:43:26.641845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.641798    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:43:26.641845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:26.641827    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:43:27.539501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:27.538989    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" event={"ID":"3639a8b0-f32f-464b-8917-a3d5656f56a2","Type":"ContainerStarted","Data":"38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c"}
Nov 01 22:43:27.539501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:27.539061    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" event={"ID":"3639a8b0-f32f-464b-8917-a3d5656f56a2","Type":"ContainerStarted","Data":"96ae85161d933ed1094039b891721734d71665733dcda1cdd54a7bbe51f11318"}
Nov 01 22:43:27.541422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:27.541395    4531 generic.go:356] "Generic (PLEG): container finished" podID="c2f14e2f-e098-4849-bbb6-d8f4e75931bd" containerID="79c0bfac5147fd36295a48c78446887d41d1e6c725f5da240ff80144634833e5" exitCode=0
Nov 01 22:43:27.541534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:27.541480    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs" event={"ID":"c2f14e2f-e098-4849-bbb6-d8f4e75931bd","Type":"ContainerDied","Data":"79c0bfac5147fd36295a48c78446887d41d1e6c725f5da240ff80144634833e5"}
Nov 01 22:43:27.541534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:27.541529    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs" event={"ID":"c2f14e2f-e098-4849-bbb6-d8f4e75931bd","Type":"ContainerStarted","Data":"f2d7b35dc4845bf419428ec5b271a2acdf7d14f9e95fdc36bc991c6491f3fd3b"}
Nov 01 22:43:27.655192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:27.655140    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podStartSLOduration=2.655127304 podStartE2EDuration="2.655127304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:27.65343819 +0000 UTC m=+1188.971088190" watchObservedRunningTime="2026-11-01 22:43:27.655127304 +0000 UTC m=+1188.972777304"
Nov 01 22:43:28.216749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:28.216689    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:28.218263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:28.218236    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 22:43:28.218376 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:28.218284    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 22:43:28.387749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:28.382596    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:28.943347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:28.943293    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:29.042171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.042095    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca\") pod \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") "
Nov 01 22:43:29.042477 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.042462    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs\" (UniqueName: \"kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs\") pod \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") "
Nov 01 22:43:29.042577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.042568    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca\") pod \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\" (UID: \"c2f14e2f-e098-4849-bbb6-d8f4e75931bd\") "
Nov 01 22:43:29.042965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.042936    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca" pod "c2f14e2f-e098-4849-bbb6-d8f4e75931bd" (UID: "c2f14e2f-e098-4849-bbb6-d8f4e75931bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:43:29.043140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.043127    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:29.044405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.044383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca" pod "c2f14e2f-e098-4849-bbb6-d8f4e75931bd" (UID: "c2f14e2f-e098-4849-bbb6-d8f4e75931bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:43:29.044470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.044418    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs" pod "c2f14e2f-e098-4849-bbb6-d8f4e75931bd" (UID: "c2f14e2f-e098-4849-bbb6-d8f4e75931bd"). InnerVolumeSpecName "kube-api-access-b6xbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:43:29.144775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.144685    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6xbs\" (UniqueName: \"kubernetes.io/projected/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-kube-api-access-b6xbs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:29.144775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.144721    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2f14e2f-e098-4849-bbb6-d8f4e75931bd-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:29.217098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.217066    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 22:43:29.217255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.217116    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 22:43:29.598081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.598047    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"
Nov 01 22:43:29.598081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.598071    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs" event={"ID":"c2f14e2f-e098-4849-bbb6-d8f4e75931bd","Type":"ContainerDied","Data":"f2d7b35dc4845bf419428ec5b271a2acdf7d14f9e95fdc36bc991c6491f3fd3b"}
Nov 01 22:43:29.598341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.598154    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2d7b35dc4845bf419428ec5b271a2acdf7d14f9e95fdc36bc991c6491f3fd3b"
Nov 01 22:43:29.776427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:29.776303    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:30.216709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:30.216680    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 22:43:30.217191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:30.216750    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 22:43:31.053907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:31.053859    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"]
Nov 01 22:43:31.104847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:31.104808    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f5149f8-2lhls"]
Nov 01 22:43:31.216632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:31.216598    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 22:43:31.216850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:31.216805    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 22:43:32.216943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:32.216909    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 22:43:32.217365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:32.216959    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 22:43:32.818250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:32.818222    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41324245-36a3-40d7-a6c8-48d60d77bacb" path="/var/lib/kubelet/pods/41324245-36a3-40d7-a6c8-48d60d77bacb/volumes"
Nov 01 22:43:33.217441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.217406    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body=
Nov 01 22:43:33.217818 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.217467    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused"
Nov 01 22:43:33.718339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.718308    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"]
Nov 01 22:43:33.718941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.718926    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c2f14e2f-e098-4849-bbb6-d8f4e75931bd" containerName="revoke-keys"
Nov 01 22:43:33.719004 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.718944    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2f14e2f-e098-4849-bbb6-d8f4e75931bd" containerName="revoke-keys"
Nov 01 22:43:33.719084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.719074    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2f14e2f-e098-4849-bbb6-d8f4e75931bd" containerName="revoke-keys"
Nov 01 22:43:33.723991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.723961    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.820672 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.820642    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c54de0\"/\"distinct-2-w0-c54de0-kserve-self-signed-certs\""
Nov 01 22:43:33.825669 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.825640    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"]
Nov 01 22:43:33.829763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.829738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.829889 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.829786    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.829889 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.829809    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.829979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.829886    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wgrs\" (UniqueName: \"kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.830016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.829994    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.830052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.830031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.932647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.932606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.933088 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.933063    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.933261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.933245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.933383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.933366    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.933503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.933488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wgrs\" (UniqueName: \"kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.933839 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.933817    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.934106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.934080    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.934666 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.934645    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.934876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.934843    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.937707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.937661    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:33.937996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:33.937977    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:34.005679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.005613    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wgrs\" (UniqueName: \"kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs\") pod \"distinct-2-w0-c54de0-kserve-8fc654759-nxvvw\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") " pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:34.011076 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.010974    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:34.040171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.040142    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:34.296074 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.295950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:34.594426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.594340    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"]
Nov 01 22:43:34.602413 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:43:34.602378    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5e89d4d_7fb0_4c36_873b_580ba2fa0468.slice/crio-4e8f1830821c0484a3c7f5ebed60a386fd1f22b5311ad8f273d9e930b9b6340a WatchSource:0}: Error finding container 4e8f1830821c0484a3c7f5ebed60a386fd1f22b5311ad8f273d9e930b9b6340a: Status 404 returned error can't find the container with id 4e8f1830821c0484a3c7f5ebed60a386fd1f22b5311ad8f273d9e930b9b6340a
Nov 01 22:43:34.613329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.613306    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:34.613651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.613630    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:34.614457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.614437    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:43:34.648392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.648364    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" event={"ID":"c5e89d4d-7fb0-4c36-873b-580ba2fa0468","Type":"ContainerStarted","Data":"4e8f1830821c0484a3c7f5ebed60a386fd1f22b5311ad8f273d9e930b9b6340a"}
Nov 01 22:43:34.962320 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:34.962292    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:35.670301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:35.670195    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" event={"ID":"c5e89d4d-7fb0-4c36-873b-580ba2fa0468","Type":"ContainerStarted","Data":"fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c"}
Nov 01 22:43:35.795841 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:35.795778    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podStartSLOduration=2.7957597180000002 podStartE2EDuration="2.795759718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:35.792708785 +0000 UTC m=+1197.110358787" watchObservedRunningTime="2026-11-01 22:43:35.795759718 +0000 UTC m=+1197.113409720"
Nov 01 22:43:36.040471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:36.040440    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:39.068518 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:39.068469    4531 scope.go:122] "RemoveContainer" containerID="3dcad0493aafefa2c9193d983f951e768e5d7eaec094f54f39f790a579fc6bbf"
Nov 01 22:43:39.084458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:39.084439    4531 scope.go:122] "RemoveContainer" containerID="2668545ba20499ee823da63ae5ca3d05be909314db1b4acc0bb742d5da49e40c"
Nov 01 22:43:40.033308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:40.033270    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:41.393060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.393029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:41.492650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.492617    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"]
Nov 01 22:43:41.498886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.498862    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.570797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.570333    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"]
Nov 01 22:43:41.593965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.593933    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.594114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.594095    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52vgp\" (UniqueName: \"kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.594187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.594148    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.601272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.601252    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:43:41.601452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.601405    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:43:41.695225 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.695203    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52vgp\" (UniqueName: \"kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.695382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.695246    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.695382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.695304    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.696087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.696068    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.698121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.698095    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.750361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.750332    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52vgp\" (UniqueName: \"kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp\") pod \"maas-api-revoke-sub-a77590dd-h4n7t\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:41.813849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:41.813824    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:42.051965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.051893    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"]
Nov 01 22:43:42.056044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.056021    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de97764e-l5df8"]
Nov 01 22:43:42.125083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.125042    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:42.200434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.200400    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"]
Nov 01 22:43:42.208932 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:43:42.206489    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e209d2_50a4_4c3f_8f4e_57c1ce47a2c7.slice/crio-b0b0a1e1211ba94706b27e8b4f552a5435bb7caa5421452748b12adcf922325c WatchSource:0}: Error finding container b0b0a1e1211ba94706b27e8b4f552a5435bb7caa5421452748b12adcf922325c: Status 404 returned error can't find the container with id b0b0a1e1211ba94706b27e8b4f552a5435bb7caa5421452748b12adcf922325c
Nov 01 22:43:42.535856 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:43:42.535816    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:43:42.749923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.749893    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t" event={"ID":"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7","Type":"ContainerStarted","Data":"2864d9f90904ee8e49f96082ff0702ff55d1bde6322692e255c9fc3a07574e8b"}
Nov 01 22:43:42.750082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.749948    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t" event={"ID":"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7","Type":"ContainerStarted","Data":"b0b0a1e1211ba94706b27e8b4f552a5435bb7caa5421452748b12adcf922325c"}
Nov 01 22:43:42.818158 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.818122    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0dd9534-ac50-43fd-bd9f-cb2be30c2e20" path="/var/lib/kubelet/pods/e0dd9534-ac50-43fd-bd9f-cb2be30c2e20/volumes"
Nov 01 22:43:42.861304 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:42.861255    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t" podStartSLOduration=1.861243084 podStartE2EDuration="1.861243084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:43:42.857066994 +0000 UTC m=+1204.174716995" watchObservedRunningTime="2026-11-01 22:43:42.861243084 +0000 UTC m=+1204.178893127"
Nov 01 22:43:43.759083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:43.759048    4531 generic.go:356] "Generic (PLEG): container finished" podID="b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" containerID="2864d9f90904ee8e49f96082ff0702ff55d1bde6322692e255c9fc3a07574e8b" exitCode=0
Nov 01 22:43:43.759456 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:43.759126    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t" event={"ID":"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7","Type":"ContainerDied","Data":"2864d9f90904ee8e49f96082ff0702ff55d1bde6322692e255c9fc3a07574e8b"}
Nov 01 22:43:44.040993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:44.040909    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:44.053322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:44.053299    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:44.778868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:44.778843    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:43:44.885633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:44.885582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:44.954706 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:44.954677    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:45.044169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.044085    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp\" (UniqueName: \"kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp\") pod \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") "
Nov 01 22:43:45.044322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.044207    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca\") pod \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") "
Nov 01 22:43:45.044322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.044263    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca\") pod \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\" (UID: \"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7\") "
Nov 01 22:43:45.044604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.044577    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca" pod "b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" (UID: "b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:43:45.044884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.044865    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:45.046075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.046055    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp" pod "b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" (UID: "b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7"). InnerVolumeSpecName "kube-api-access-52vgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:43:45.046148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.046102    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca" pod "b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" (UID: "b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:43:45.145841 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.145805    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52vgp\" (UniqueName: \"kubernetes.io/projected/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-kube-api-access-52vgp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:45.145841 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.145839    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:43:45.777288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.777256    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"
Nov 01 22:43:45.777455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.777280    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t" event={"ID":"b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7","Type":"ContainerDied","Data":"b0b0a1e1211ba94706b27e8b4f552a5435bb7caa5421452748b12adcf922325c"}
Nov 01 22:43:45.777455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:45.777351    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0b0a1e1211ba94706b27e8b4f552a5435bb7caa5421452748b12adcf922325c"
Nov 01 22:43:49.802453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:49.802350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:54.551857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:54.551809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:58.635563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:58.635525    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:58.864617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:58.864578    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:59.218293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.218257    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"]
Nov 01 22:43:59.219192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.219175    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" containerName="revoke-keys"
Nov 01 22:43:59.219254 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.219195    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" containerName="revoke-keys"
Nov 01 22:43:59.219388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.219377    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" containerName="revoke-keys"
Nov 01 22:43:59.223417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.223396    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309237    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309372    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309426    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309456    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsx8\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309711    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309806    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.309959 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309942    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.310006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.309975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411487    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411543    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411570    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsx8\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411669    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411909 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411698    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411909 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.411909 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411852    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.412066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.411915    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.412066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.412018    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.412181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.412077    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.412181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.412160    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.412333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.412314    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.412542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.412518    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.413792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.413773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.414197 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.414179    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.423646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.423622    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"]
Nov 01 22:43:59.454621 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.454584    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:43:59.466031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.465962    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.474086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.474024    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsx8\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.477974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.477947    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"]
Nov 01 22:43:59.489347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.489325    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.538629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.538386    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"]
Nov 01 22:43:59.538876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.538857    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:43:59.614331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614222    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.614720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614419    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.614720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614494    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.614720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614557    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.614720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614588    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.614720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614689    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.615127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614756    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.615127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.615127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.614803    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98m5\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.715946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.715919    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.715998    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716018    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716045    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716061    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716138    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716197    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716220    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716250    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98m5\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716593    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716694    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.716878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.716708    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.717255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.717234    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.718366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.718347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.718808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.718793    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.770238 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.770164    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.770722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.770699    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98m5\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5\") pod \"e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") " pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.815857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.815831    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:43:59.991974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.991941    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"]
Nov 01 22:43:59.993271 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:43:59.993240    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb49781_84b7_4132_9072_d40a0d545345.slice/crio-776e64585e88ddcefc94835dd6c461ae3ca459ee2f2e543f09ad28d04c8501ee WatchSource:0}: Error finding container 776e64585e88ddcefc94835dd6c461ae3ca459ee2f2e543f09ad28d04c8501ee: Status 404 returned error can't find the container with id 776e64585e88ddcefc94835dd6c461ae3ca459ee2f2e543f09ad28d04c8501ee
Nov 01 22:43:59.996051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.996014    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:43:59.996140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.996095    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:43:59.996180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:43:59.996136    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:00.209826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.209797    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"]
Nov 01 22:44:00.211664 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:00.211638    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9bd15c1_bd84_460b_b07a_0e77a6aed64e.slice/crio-4dd585a1dbd907665a480aee9bf59c7bf1e5295eaeaf5e3e8ed2adfbcc468ac5 WatchSource:0}: Error finding container 4dd585a1dbd907665a480aee9bf59c7bf1e5295eaeaf5e3e8ed2adfbcc468ac5: Status 404 returned error can't find the container with id 4dd585a1dbd907665a480aee9bf59c7bf1e5295eaeaf5e3e8ed2adfbcc468ac5
Nov 01 22:44:00.214403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.214369    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:00.214522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.214440    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:00.214522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.214487    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:00.913236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.913203    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" event={"ID":"ddb49781-84b7-4132-9072-d40a0d545345","Type":"ContainerStarted","Data":"96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a"}
Nov 01 22:44:00.913630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.913255    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" event={"ID":"ddb49781-84b7-4132-9072-d40a0d545345","Type":"ContainerStarted","Data":"776e64585e88ddcefc94835dd6c461ae3ca459ee2f2e543f09ad28d04c8501ee"}
Nov 01 22:44:00.916008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.915980    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" event={"ID":"e9bd15c1-bd84-460b-b07a-0e77a6aed64e","Type":"ContainerStarted","Data":"30c33a36db19f52ba58ce944bce8b129efc08ca3221fc27c54383f4e8497b788"}
Nov 01 22:44:00.916133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:00.916023    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" event={"ID":"e9bd15c1-bd84-460b-b07a-0e77a6aed64e","Type":"ContainerStarted","Data":"4dd585a1dbd907665a480aee9bf59c7bf1e5295eaeaf5e3e8ed2adfbcc468ac5"}
Nov 01 22:44:01.034235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.034184    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podStartSLOduration=2.034169981 podStartE2EDuration="2.034169981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:01.027987413 +0000 UTC m=+1222.345637413" watchObservedRunningTime="2026-11-01 22:44:01.034169981 +0000 UTC m=+1222.351819982"
Nov 01 22:44:01.150746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.149780    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podStartSLOduration=2.149759391 podStartE2EDuration="2.149759391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:01.142815372 +0000 UTC m=+1222.460465381" watchObservedRunningTime="2026-11-01 22:44:01.149759391 +0000 UTC m=+1222.467409394"
Nov 01 22:44:01.178801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.177947    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"]
Nov 01 22:44:01.198751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.198058    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.247462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.247422    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"]
Nov 01 22:44:01.248038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.248014    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:01.248423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.248397    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:01.349235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.349192    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4zp\" (UniqueName: \"kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.349707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.349682    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.350229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.350208    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.451851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.451821    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4zp\" (UniqueName: \"kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.452002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.451916    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.452002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.451941    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.452590 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.452567    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.454316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.454297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.519535 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.519501    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4zp\" (UniqueName: \"kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp\") pod \"maas-api-revoke-sub-aceb0f25-pfjfd\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.542156 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.542124    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:44:01.543876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.543843    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 22:44:01.544028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.543893    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 22:44:01.547277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.547254    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:01.829749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.827656    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 22:44:01.830117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.830085    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 22:44:01.831556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:01.831537    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:44:02.120934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.108874    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"]
Nov 01 22:44:02.125370 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:02.125305    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47901d08_5ff6_492e_94e6_47a3addec19e.slice/crio-b7834a435eee3772752866b6e30dc8f42fb6b076f60d69fa3fa7c2f656285e91 WatchSource:0}: Error finding container b7834a435eee3772752866b6e30dc8f42fb6b076f60d69fa3fa7c2f656285e91: Status 404 returned error can't find the container with id b7834a435eee3772752866b6e30dc8f42fb6b076f60d69fa3fa7c2f656285e91
Nov 01 22:44:02.539933 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.539905    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 22:44:02.540087 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.539952    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 22:44:02.819758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.819365    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 22:44:02.819758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.819421    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 22:44:02.947704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.947630    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd" event={"ID":"47901d08-5ff6-492e-94e6-47a3addec19e","Type":"ContainerStarted","Data":"7bbbbbe1dfbb15af1e1f76aac33a8c0fe92b414f083d9761aaa8df4bd7be71ae"}
Nov 01 22:44:02.947704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:02.947680    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd" event={"ID":"47901d08-5ff6-492e-94e6-47a3addec19e","Type":"ContainerStarted","Data":"b7834a435eee3772752866b6e30dc8f42fb6b076f60d69fa3fa7c2f656285e91"}
Nov 01 22:44:03.116804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.116452    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd" podStartSLOduration=2.116418953 podStartE2EDuration="2.116418953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:03.095180665 +0000 UTC m=+1224.412830666" watchObservedRunningTime="2026-11-01 22:44:03.116418953 +0000 UTC m=+1224.434068957"
Nov 01 22:44:03.540245 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.540210    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 22:44:03.540632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.540257    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 22:44:03.818752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.818147    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 22:44:03.818752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.818213    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 22:44:03.957971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.957940    4531 generic.go:356] "Generic (PLEG): container finished" podID="47901d08-5ff6-492e-94e6-47a3addec19e" containerID="7bbbbbe1dfbb15af1e1f76aac33a8c0fe92b414f083d9761aaa8df4bd7be71ae" exitCode=0
Nov 01 22:44:03.958137 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:03.958028    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd" event={"ID":"47901d08-5ff6-492e-94e6-47a3addec19e","Type":"ContainerDied","Data":"7bbbbbe1dfbb15af1e1f76aac33a8c0fe92b414f083d9761aaa8df4bd7be71ae"}
Nov 01 22:44:04.539902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:04.539872    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 22:44:04.540080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:04.539924    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 22:44:04.581410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:04.581189    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:04.820980 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:04.820895    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 22:44:04.834055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:04.821940    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 22:44:05.411848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.411010    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:05.484756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.484707    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca\") pod \"47901d08-5ff6-492e-94e6-47a3addec19e\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") "
Nov 01 22:44:05.485518 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.485496    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp\" (UniqueName: \"kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp\") pod \"47901d08-5ff6-492e-94e6-47a3addec19e\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") "
Nov 01 22:44:05.485751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.485719    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca\") pod \"47901d08-5ff6-492e-94e6-47a3addec19e\" (UID: \"47901d08-5ff6-492e-94e6-47a3addec19e\") "
Nov 01 22:44:05.491772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.485269    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca" pod "47901d08-5ff6-492e-94e6-47a3addec19e" (UID: "47901d08-5ff6-492e-94e6-47a3addec19e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:05.532422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.531097    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca" pod "47901d08-5ff6-492e-94e6-47a3addec19e" (UID: "47901d08-5ff6-492e-94e6-47a3addec19e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:05.532855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.532825    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp" pod "47901d08-5ff6-492e-94e6-47a3addec19e" (UID: "47901d08-5ff6-492e-94e6-47a3addec19e"). InnerVolumeSpecName "kube-api-access-rc4zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:05.540122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.540088    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 22:44:05.540238 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.540143    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 22:44:05.596750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.587775    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:05.596750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.587815    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47901d08-5ff6-492e-94e6-47a3addec19e-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:05.596750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.587834    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc4zp\" (UniqueName: \"kubernetes.io/projected/47901d08-5ff6-492e-94e6-47a3addec19e-kube-api-access-rc4zp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:05.653187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.653149    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"]
Nov 01 22:44:05.655241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.655208    4531 cpu_manager.go:414] "containerMap: removing container" podUID="47901d08-5ff6-492e-94e6-47a3addec19e" containerName="revoke-keys"
Nov 01 22:44:05.655241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.655238    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47901d08-5ff6-492e-94e6-47a3addec19e" containerName="revoke-keys"
Nov 01 22:44:05.655432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.655394    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="47901d08-5ff6-492e-94e6-47a3addec19e" containerName="revoke-keys"
Nov 01 22:44:05.664033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.663969    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.741902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.738946    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"]
Nov 01 22:44:05.809068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.809019    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.834426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.833284    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4bcr\" (UniqueName: \"kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.834426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.833482    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.834426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.829146    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 22:44:05.834426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.834297    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 22:44:05.935369 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.935287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.935369 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.935366    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.935605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.935496    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4bcr\" (UniqueName: \"kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.936339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.936315    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:05.938023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:05.938004    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:06.007693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.007662    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4bcr\" (UniqueName: \"kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr\") pod \"maas-api-revoke-sub-b5108a13-kz4nm\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:06.033138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.033105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd" event={"ID":"47901d08-5ff6-492e-94e6-47a3addec19e","Type":"ContainerDied","Data":"b7834a435eee3772752866b6e30dc8f42fb6b076f60d69fa3fa7c2f656285e91"}
Nov 01 22:44:06.033367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.033354    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7834a435eee3772752866b6e30dc8f42fb6b076f60d69fa3fa7c2f656285e91"
Nov 01 22:44:06.033478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.033468    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"
Nov 01 22:44:06.052331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.052307    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:06.540743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.540650    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused" start-of-body=
Nov 01 22:44:06.540743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.540709    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": dial tcp 10.133.0.110:15021: connect: connection refused"
Nov 01 22:44:06.588999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.588962    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"]
Nov 01 22:44:06.592761 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:06.590821    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ced4ee3_1e13_44f8_8f93_75226f143d62.slice/crio-ba6d8d1349c548404a307158e70cb2abc91a909d56f31774702d5f807b1bbdd5 WatchSource:0}: Error finding container ba6d8d1349c548404a307158e70cb2abc91a909d56f31774702d5f807b1bbdd5: Status 404 returned error can't find the container with id ba6d8d1349c548404a307158e70cb2abc91a909d56f31774702d5f807b1bbdd5
Nov 01 22:44:06.816393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.816300    4531 patch_prober.go:29] interesting pod/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused" start-of-body=
Nov 01 22:44:06.816393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:06.816365    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.111:15021/healthz/ready\": dial tcp 10.133.0.111:15021: connect: connection refused"
Nov 01 22:44:07.043961 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:07.043923    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm" event={"ID":"5ced4ee3-1e13-44f8-8f93-75226f143d62","Type":"ContainerStarted","Data":"dcf076963487b0349faed467ebe7a2fc6ede330f63396bae9d58b6e139db3a88"}
Nov 01 22:44:07.044113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:07.043991    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm" event={"ID":"5ced4ee3-1e13-44f8-8f93-75226f143d62","Type":"ContainerStarted","Data":"ba6d8d1349c548404a307158e70cb2abc91a909d56f31774702d5f807b1bbdd5"}
Nov 01 22:44:07.157197 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:07.157101    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm" podStartSLOduration=2.157087692 podStartE2EDuration="2.157087692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:07.151592652 +0000 UTC m=+1228.469242656" watchObservedRunningTime="2026-11-01 22:44:07.157087692 +0000 UTC m=+1228.474737693"
Nov 01 22:44:07.930232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:07.930198    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:44:07.931165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:07.931137    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:44:07.931936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:07.931914    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 22:44:08.059633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:08.059540    4531 generic.go:356] "Generic (PLEG): container finished" podID="5ced4ee3-1e13-44f8-8f93-75226f143d62" containerID="dcf076963487b0349faed467ebe7a2fc6ede330f63396bae9d58b6e139db3a88" exitCode=0
Nov 01 22:44:08.059827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:08.059623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm" event={"ID":"5ced4ee3-1e13-44f8-8f93-75226f143d62","Type":"ContainerDied","Data":"dcf076963487b0349faed467ebe7a2fc6ede330f63396bae9d58b6e139db3a88"}
Nov 01 22:44:08.252335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:08.252307    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:44:08.252789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:08.252765    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:44:08.254031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:08.254013    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 22:44:09.447066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.447036    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:09.550260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.550228    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr\" (UniqueName: \"kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr\") pod \"5ced4ee3-1e13-44f8-8f93-75226f143d62\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") "
Nov 01 22:44:09.551199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.551175    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca\") pod \"5ced4ee3-1e13-44f8-8f93-75226f143d62\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") "
Nov 01 22:44:09.551845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.551826    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca\") pod \"5ced4ee3-1e13-44f8-8f93-75226f143d62\" (UID: \"5ced4ee3-1e13-44f8-8f93-75226f143d62\") "
Nov 01 22:44:09.552940 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.552292    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca" pod "5ced4ee3-1e13-44f8-8f93-75226f143d62" (UID: "5ced4ee3-1e13-44f8-8f93-75226f143d62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:09.563830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.561879    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca" pod "5ced4ee3-1e13-44f8-8f93-75226f143d62" (UID: "5ced4ee3-1e13-44f8-8f93-75226f143d62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:09.725319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.725177    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:09.738691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.738630    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:09.738691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.738664    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ced4ee3-1e13-44f8-8f93-75226f143d62-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:09.940748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.923931    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr" pod "5ced4ee3-1e13-44f8-8f93-75226f143d62" (UID: "5ced4ee3-1e13-44f8-8f93-75226f143d62"). InnerVolumeSpecName "kube-api-access-x4bcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:10.001800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:09.982840    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4bcr\" (UniqueName: \"kubernetes.io/projected/5ced4ee3-1e13-44f8-8f93-75226f143d62-kube-api-access-x4bcr\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:10.167751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:10.166681    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"
Nov 01 22:44:10.189751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:10.182952    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm" event={"ID":"5ced4ee3-1e13-44f8-8f93-75226f143d62","Type":"ContainerDied","Data":"ba6d8d1349c548404a307158e70cb2abc91a909d56f31774702d5f807b1bbdd5"}
Nov 01 22:44:10.189751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:10.183028    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6d8d1349c548404a307158e70cb2abc91a909d56f31774702d5f807b1bbdd5"
Nov 01 22:44:11.044964 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:11.044921    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:12.586540 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:44:12.586504    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:44:13.150697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.150661    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"]
Nov 01 22:44:13.151644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.151620    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5ced4ee3-1e13-44f8-8f93-75226f143d62" containerName="revoke-keys"
Nov 01 22:44:13.151644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.151642    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ced4ee3-1e13-44f8-8f93-75226f143d62" containerName="revoke-keys"
Nov 01 22:44:13.151891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.151876    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ced4ee3-1e13-44f8-8f93-75226f143d62" containerName="revoke-keys"
Nov 01 22:44:13.161942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.161900    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.217759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.217431    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"]
Nov 01 22:44:13.258222 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.256896    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:13.258222 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.257823    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:13.271925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.271886    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.272081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.271987    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.272081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.272051    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv5xh\" (UniqueName: \"kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.372921 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.372880    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.373106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.372961    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.373106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.373027    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv5xh\" (UniqueName: \"kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.373637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.373612    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.375209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.375180    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.429901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.429836    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv5xh\" (UniqueName: \"kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh\") pod \"maas-api-revoke-sub-aaf3fd38-m7ll7\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.483271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.483235    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:13.910436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:13.910404    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"]
Nov 01 22:44:13.911604 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:13.911573    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8d1ac0_5ef8_42ec_8dfc_6bac02178cef.slice/crio-0a55fd38740590d4a2def4e95c558f23f271bfe2dc0fd0f4f379f3d03806dd06 WatchSource:0}: Error finding container 0a55fd38740590d4a2def4e95c558f23f271bfe2dc0fd0f4f379f3d03806dd06: Status 404 returned error can't find the container with id 0a55fd38740590d4a2def4e95c558f23f271bfe2dc0fd0f4f379f3d03806dd06
Nov 01 22:44:14.222496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:14.222464    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7" event={"ID":"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef","Type":"ContainerStarted","Data":"0a55fd38740590d4a2def4e95c558f23f271bfe2dc0fd0f4f379f3d03806dd06"}
Nov 01 22:44:15.159007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:15.158969    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:15.234406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:15.234373    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" containerID="e9f6efc3a3da9b15846c8d82e376fb9a90b38627df47d6190342149f69ec9365" exitCode=0
Nov 01 22:44:15.234586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:15.234448    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7" event={"ID":"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef","Type":"ContainerDied","Data":"e9f6efc3a3da9b15846c8d82e376fb9a90b38627df47d6190342149f69ec9365"}
Nov 01 22:44:16.391173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.391143    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:16.410221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.410198    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca\") pod \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") "
Nov 01 22:44:16.410334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.410309    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh\" (UniqueName: \"kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh\") pod \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") "
Nov 01 22:44:16.412176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.412148    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca" pod "ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" (UID: "ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:16.412305 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.412284    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh" pod "ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" (UID: "ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef"). InnerVolumeSpecName "kube-api-access-sv5xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:16.511409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.511349    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca\") pod \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\" (UID: \"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef\") "
Nov 01 22:44:16.511690 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.511668    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca" pod "ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" (UID: "ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:16.511838 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.511821    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:16.511897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.511844    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:16.511897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:16.511872    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv5xh\" (UniqueName: \"kubernetes.io/projected/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef-kube-api-access-sv5xh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:17.251890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:17.251859    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"
Nov 01 22:44:17.252033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:17.251895    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7" event={"ID":"ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef","Type":"ContainerDied","Data":"0a55fd38740590d4a2def4e95c558f23f271bfe2dc0fd0f4f379f3d03806dd06"}
Nov 01 22:44:17.252033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:17.251967    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a55fd38740590d4a2def4e95c558f23f271bfe2dc0fd0f4f379f3d03806dd06"
Nov 01 22:44:20.042684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:20.042368    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:22.487937 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.487902    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"]
Nov 01 22:44:22.488510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.488497    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" containerName="revoke-keys"
Nov 01 22:44:22.488555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.488511    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" containerName="revoke-keys"
Nov 01 22:44:22.488646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.488637    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" containerName="revoke-keys"
Nov 01 22:44:22.491975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.491960    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.546555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.546526    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"]
Nov 01 22:44:22.584408 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.584380    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgwf\" (UniqueName: \"kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.584617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.584590    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.584692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.584675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.587342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.587313    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:22.587342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.587316    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:22.685324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.685299    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.685454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.685362    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgwf\" (UniqueName: \"kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.685454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.685450    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.686088 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.686068    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.687692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.687673    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.739667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.739615    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgwf\" (UniqueName: \"kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf\") pod \"maas-api-revoke-sub-d2e6d593-vv69z\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:22.806379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:22.806355    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:23.203890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:23.203862    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"]
Nov 01 22:44:23.207346 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:23.207318    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d41d4e_078f_4809_a3c2_1a96e5fc3948.slice/crio-9ed79ec1a6798fc1e76bd54ece1f249eaf055c73ad47d15eb2c2f5ce8f89e7bb WatchSource:0}: Error finding container 9ed79ec1a6798fc1e76bd54ece1f249eaf055c73ad47d15eb2c2f5ce8f89e7bb: Status 404 returned error can't find the container with id 9ed79ec1a6798fc1e76bd54ece1f249eaf055c73ad47d15eb2c2f5ce8f89e7bb
Nov 01 22:44:23.305847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:23.305814    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z" event={"ID":"30d41d4e-078f-4809-a3c2-1a96e5fc3948","Type":"ContainerStarted","Data":"9ed79ec1a6798fc1e76bd54ece1f249eaf055c73ad47d15eb2c2f5ce8f89e7bb"}
Nov 01 22:44:24.315882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:24.315849    4531 generic.go:356] "Generic (PLEG): container finished" podID="30d41d4e-078f-4809-a3c2-1a96e5fc3948" containerID="ac14cef12d07c4e5a31933c7a8a9b65d35b84456c6e2d17b7e8243877e691adf" exitCode=0
Nov 01 22:44:24.316257 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:24.315929    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z" event={"ID":"30d41d4e-078f-4809-a3c2-1a96e5fc3948","Type":"ContainerDied","Data":"ac14cef12d07c4e5a31933c7a8a9b65d35b84456c6e2d17b7e8243877e691adf"}
Nov 01 22:44:24.855022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:24.854971    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:25.498465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.498437    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:25.644507 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.644434    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf\" (UniqueName: \"kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf\") pod \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") "
Nov 01 22:44:25.644507 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.644497    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca\") pod \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") "
Nov 01 22:44:25.644764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.644564    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca\") pod \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\" (UID: \"30d41d4e-078f-4809-a3c2-1a96e5fc3948\") "
Nov 01 22:44:25.644941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.644912    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca" pod "30d41d4e-078f-4809-a3c2-1a96e5fc3948" (UID: "30d41d4e-078f-4809-a3c2-1a96e5fc3948"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:25.645218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.645197    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:25.646697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.646676    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf" pod "30d41d4e-078f-4809-a3c2-1a96e5fc3948" (UID: "30d41d4e-078f-4809-a3c2-1a96e5fc3948"). InnerVolumeSpecName "kube-api-access-xfgwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:25.647046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.647023    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca" pod "30d41d4e-078f-4809-a3c2-1a96e5fc3948" (UID: "30d41d4e-078f-4809-a3c2-1a96e5fc3948"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:25.746364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.746339    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30d41d4e-078f-4809-a3c2-1a96e5fc3948-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:25.746364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:25.746364    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfgwf\" (UniqueName: \"kubernetes.io/projected/30d41d4e-078f-4809-a3c2-1a96e5fc3948-kube-api-access-xfgwf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:26.334641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:26.334616    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"
Nov 01 22:44:26.334830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:26.334658    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z" event={"ID":"30d41d4e-078f-4809-a3c2-1a96e5fc3948","Type":"ContainerDied","Data":"9ed79ec1a6798fc1e76bd54ece1f249eaf055c73ad47d15eb2c2f5ce8f89e7bb"}
Nov 01 22:44:26.334830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:26.334703    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ed79ec1a6798fc1e76bd54ece1f249eaf055c73ad47d15eb2c2f5ce8f89e7bb"
Nov 01 22:44:28.049782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:28.049747    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"]
Nov 01 22:44:28.095319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:28.095289    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2310dbb2-wv8ps"]
Nov 01 22:44:28.819837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:28.819806    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71e3b5c4-fa38-45b1-9861-9ce4dfbecccf" path="/var/lib/kubelet/pods/71e3b5c4-fa38-45b1-9861-9ce4dfbecccf/volumes"
Nov 01 22:44:29.763026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:29.762974    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:30.056998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.056923    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"]
Nov 01 22:44:30.057901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.057885    4531 cpu_manager.go:414] "containerMap: removing container" podUID="30d41d4e-078f-4809-a3c2-1a96e5fc3948" containerName="revoke-keys"
Nov 01 22:44:30.057962 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.057904    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d41d4e-078f-4809-a3c2-1a96e5fc3948" containerName="revoke-keys"
Nov 01 22:44:30.058039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.058029    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d41d4e-078f-4809-a3c2-1a96e5fc3948" containerName="revoke-keys"
Nov 01 22:44:30.062381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.062360    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.116660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.115604    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"]
Nov 01 22:44:30.156489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.156459    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:30.156687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.156555    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:30.212551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.212513    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.212744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.212581    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glh86\" (UniqueName: \"kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.212744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.212649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.314174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.314106    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.314174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.314156    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glh86\" (UniqueName: \"kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.314174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.314175    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.314899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.314870    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.316588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.316564    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.367105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.367080    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glh86\" (UniqueName: \"kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86\") pod \"maas-api-revoke-sub-fd59a338-r2cvr\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.376903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.376875    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:30.756632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:30.756607    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"]
Nov 01 22:44:30.759454 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:30.759429    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69837d8_c248_4181_91bf_b39cfd9791bd.slice/crio-83a0563fa9f7468a39587d413dab8c1e4760542373739411652e5f6a3e8d4049 WatchSource:0}: Error finding container 83a0563fa9f7468a39587d413dab8c1e4760542373739411652e5f6a3e8d4049: Status 404 returned error can't find the container with id 83a0563fa9f7468a39587d413dab8c1e4760542373739411652e5f6a3e8d4049
Nov 01 22:44:31.052890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:31.052804    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"]
Nov 01 22:44:31.059643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:31.059622    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-319bbba5-hf7j4"]
Nov 01 22:44:31.380417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:31.380326    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr" event={"ID":"a69837d8-c248-4181-91bf-b39cfd9791bd","Type":"ContainerStarted","Data":"ac46db010d04a51727bbdf077bfa0d83cad117e3f314b1f6ac959b9e17deffd2"}
Nov 01 22:44:31.380417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:31.380392    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr" event={"ID":"a69837d8-c248-4181-91bf-b39cfd9791bd","Type":"ContainerStarted","Data":"83a0563fa9f7468a39587d413dab8c1e4760542373739411652e5f6a3e8d4049"}
Nov 01 22:44:31.498183 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:31.498135    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr" podStartSLOduration=1.49812185 podStartE2EDuration="1.49812185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:31.494037486 +0000 UTC m=+1252.811687489" watchObservedRunningTime="2026-11-01 22:44:31.49812185 +0000 UTC m=+1252.815771851"
Nov 01 22:44:32.390432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:32.390396    4531 generic.go:356] "Generic (PLEG): container finished" podID="a69837d8-c248-4181-91bf-b39cfd9791bd" containerID="ac46db010d04a51727bbdf077bfa0d83cad117e3f314b1f6ac959b9e17deffd2" exitCode=0
Nov 01 22:44:32.390953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:32.390474    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr" event={"ID":"a69837d8-c248-4181-91bf-b39cfd9791bd","Type":"ContainerDied","Data":"ac46db010d04a51727bbdf077bfa0d83cad117e3f314b1f6ac959b9e17deffd2"}
Nov 01 22:44:32.817851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:32.817821    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a86243-8729-496f-a291-95b8d37ff93a" path="/var/lib/kubelet/pods/c3a86243-8729-496f-a291-95b8d37ff93a/volumes"
Nov 01 22:44:33.580126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.580095    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:33.651906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.651879    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca\") pod \"a69837d8-c248-4181-91bf-b39cfd9791bd\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") "
Nov 01 22:44:33.652048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.651918    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86\" (UniqueName: \"kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86\") pod \"a69837d8-c248-4181-91bf-b39cfd9791bd\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") "
Nov 01 22:44:33.652048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.652015    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca\") pod \"a69837d8-c248-4181-91bf-b39cfd9791bd\" (UID: \"a69837d8-c248-4181-91bf-b39cfd9791bd\") "
Nov 01 22:44:33.652359 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.652327    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca" pod "a69837d8-c248-4181-91bf-b39cfd9791bd" (UID: "a69837d8-c248-4181-91bf-b39cfd9791bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:33.653910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.653886    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca" pod "a69837d8-c248-4181-91bf-b39cfd9791bd" (UID: "a69837d8-c248-4181-91bf-b39cfd9791bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:33.654018 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.653993    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86" pod "a69837d8-c248-4181-91bf-b39cfd9791bd" (UID: "a69837d8-c248-4181-91bf-b39cfd9791bd"). InnerVolumeSpecName "kube-api-access-glh86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:33.754021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.753990    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:33.754021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.754018    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a69837d8-c248-4181-91bf-b39cfd9791bd-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:33.754192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:33.754028    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glh86\" (UniqueName: \"kubernetes.io/projected/a69837d8-c248-4181-91bf-b39cfd9791bd-kube-api-access-glh86\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:34.412331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:34.412304    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"
Nov 01 22:44:34.412496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:34.412340    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr" event={"ID":"a69837d8-c248-4181-91bf-b39cfd9791bd","Type":"ContainerDied","Data":"83a0563fa9f7468a39587d413dab8c1e4760542373739411652e5f6a3e8d4049"}
Nov 01 22:44:34.412496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:34.412387    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83a0563fa9f7468a39587d413dab8c1e4760542373739411652e5f6a3e8d4049"
Nov 01 22:44:34.576118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:34.576086    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:34.764315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:34.764285    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:34.956211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:34.956178    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:35.496077 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.496036    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"]
Nov 01 22:44:35.496943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.496920    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a69837d8-c248-4181-91bf-b39cfd9791bd" containerName="revoke-keys"
Nov 01 22:44:35.496943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.496941    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a69837d8-c248-4181-91bf-b39cfd9791bd" containerName="revoke-keys"
Nov 01 22:44:35.497131 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.497062    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a69837d8-c248-4181-91bf-b39cfd9791bd" containerName="revoke-keys"
Nov 01 22:44:35.500904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.500888    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.552181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.552153    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"]
Nov 01 22:44:35.576434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.576380    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46hgb\" (UniqueName: \"kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.576689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.576662    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.576865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.576776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.592619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.592592    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:44:35.592745 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.592623    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:44:35.677486 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.677467    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46hgb\" (UniqueName: \"kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.677595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.677574    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.677643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.677629    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.678318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.678297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.679969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.679945    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.737082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.737057    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46hgb\" (UniqueName: \"kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb\") pod \"maas-api-revoke-sub-22fd8c89-62gt9\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:35.815290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:35.815225    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:36.204762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:36.201447    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"]
Nov 01 22:44:36.209519 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:36.209481    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea68789_157d_477e_acda_10b572466456.slice/crio-33c1ce38f03b0d7d668e4583e22ada80dff5ec441bcfbce61d6ac4bac2968911 WatchSource:0}: Error finding container 33c1ce38f03b0d7d668e4583e22ada80dff5ec441bcfbce61d6ac4bac2968911: Status 404 returned error can't find the container with id 33c1ce38f03b0d7d668e4583e22ada80dff5ec441bcfbce61d6ac4bac2968911
Nov 01 22:44:36.428818 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:36.428786    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9" event={"ID":"3ea68789-157d-477e-acda-10b572466456","Type":"ContainerStarted","Data":"33c1ce38f03b0d7d668e4583e22ada80dff5ec441bcfbce61d6ac4bac2968911"}
Nov 01 22:44:37.440052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:37.439936    4531 generic.go:356] "Generic (PLEG): container finished" podID="3ea68789-157d-477e-acda-10b572466456" containerID="0f335d3c0fa82caa710e4cbe8b08b6570bbe2346ba0d3ea10b84c8d551960ab1" exitCode=0
Nov 01 22:44:37.440052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:37.440021    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9" event={"ID":"3ea68789-157d-477e-acda-10b572466456","Type":"ContainerDied","Data":"0f335d3c0fa82caa710e4cbe8b08b6570bbe2346ba0d3ea10b84c8d551960ab1"}
Nov 01 22:44:38.513318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.513290    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"]
Nov 01 22:44:38.519559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.518423    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.578945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.578914    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"]
Nov 01 22:44:38.628768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.628737    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6grm\" (UniqueName: \"kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.628942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.628917    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.629033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.629015    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.639332 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.639313    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:38.729775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.729697    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca\") pod \"3ea68789-157d-477e-acda-10b572466456\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") "
Nov 01 22:44:38.729775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.729770    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca\") pod \"3ea68789-157d-477e-acda-10b572466456\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") "
Nov 01 22:44:38.729926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.729875    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb\" (UniqueName: \"kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb\") pod \"3ea68789-157d-477e-acda-10b572466456\" (UID: \"3ea68789-157d-477e-acda-10b572466456\") "
Nov 01 22:44:38.730162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.730131    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca" pod "3ea68789-157d-477e-acda-10b572466456" (UID: "3ea68789-157d-477e-acda-10b572466456"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:38.730263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.730139    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.730263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.730243    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.730377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.730334    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6grm\" (UniqueName: \"kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.730434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.730425    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea68789-157d-477e-acda-10b572466456-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:38.730915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.730889    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.731795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.731768    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca" pod "3ea68789-157d-477e-acda-10b572466456" (UID: "3ea68789-157d-477e-acda-10b572466456"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:38.732057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.732032    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb" pod "3ea68789-157d-477e-acda-10b572466456" (UID: "3ea68789-157d-477e-acda-10b572466456"). InnerVolumeSpecName "kube-api-access-46hgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:38.732450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.732433    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.786175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.786148    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6grm\" (UniqueName: \"kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm\") pod \"maas-api-revoke-sub-d004d649-92bh9\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:38.831536 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.831510    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea68789-157d-477e-acda-10b572466456-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:38.831653 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.831538    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46hgb\" (UniqueName: \"kubernetes.io/projected/3ea68789-157d-477e-acda-10b572466456-kube-api-access-46hgb\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:38.847629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:38.847605    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:39.122756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.122670    4531 scope.go:122] "RemoveContainer" containerID="e01f5b5c887aa4680dcc2ac85fe348e32a83013ce5415fe0851ee65036a394ef"
Nov 01 22:44:39.133576 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.133554    4531 scope.go:122] "RemoveContainer" containerID="0a5bef3d495d10d5dd59bf11ece6d4dcb3b532018692458a10b05f7b2fa2832f"
Nov 01 22:44:39.143302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.143281    4531 scope.go:122] "RemoveContainer" containerID="b79baee697971079b3dfc6b0bd235ab94d97ee8ab2f4fd3fbda464a4f22686e8"
Nov 01 22:44:39.153377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.153358    4531 scope.go:122] "RemoveContainer" containerID="ba64f6acfbd3d2c814f1006bd0c8ac9f200036e85ff4e18aad9d10e8fdd9534e"
Nov 01 22:44:39.243591 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.243570    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"]
Nov 01 22:44:39.246210 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:39.246182    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ab4e74_c70b_4dd5_823f_d208cf8fb02e.slice/crio-7d9415b67e0764c2a82c39c438a63fb377dcd186ffb70a4abdd3a538e1fc9fdd WatchSource:0}: Error finding container 7d9415b67e0764c2a82c39c438a63fb377dcd186ffb70a4abdd3a538e1fc9fdd: Status 404 returned error can't find the container with id 7d9415b67e0764c2a82c39c438a63fb377dcd186ffb70a4abdd3a538e1fc9fdd
Nov 01 22:44:39.457296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.457259    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9" event={"ID":"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e","Type":"ContainerStarted","Data":"7d9415b67e0764c2a82c39c438a63fb377dcd186ffb70a4abdd3a538e1fc9fdd"}
Nov 01 22:44:39.458899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.458875    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"
Nov 01 22:44:39.458899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.458887    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9" event={"ID":"3ea68789-157d-477e-acda-10b572466456","Type":"ContainerDied","Data":"33c1ce38f03b0d7d668e4583e22ada80dff5ec441bcfbce61d6ac4bac2968911"}
Nov 01 22:44:39.459059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.458932    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33c1ce38f03b0d7d668e4583e22ada80dff5ec441bcfbce61d6ac4bac2968911"
Nov 01 22:44:39.556509 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:39.556474    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:40.474283 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:40.474248    4531 generic.go:356] "Generic (PLEG): container finished" podID="c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" containerID="8c8e746de7a4d03543094db93b88b723a73ac361ff4cd400c033e5d7d258bbb8" exitCode=0
Nov 01 22:44:40.474475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:40.474394    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9" event={"ID":"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e","Type":"ContainerDied","Data":"8c8e746de7a4d03543094db93b88b723a73ac361ff4cd400c033e5d7d258bbb8"}
Nov 01 22:44:41.281042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.281008    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"]
Nov 01 22:44:41.281908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.281884    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3ea68789-157d-477e-acda-10b572466456" containerName="revoke-keys"
Nov 01 22:44:41.281908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.281905    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea68789-157d-477e-acda-10b572466456" containerName="revoke-keys"
Nov 01 22:44:41.282092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.282028    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea68789-157d-477e-acda-10b572466456" containerName="revoke-keys"
Nov 01 22:44:41.285903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.285884    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.327714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.327683    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"]
Nov 01 22:44:41.332826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.332805    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"]
Nov 01 22:44:41.332971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.332936    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.337188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.337165    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.364834 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.364812    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pntrq\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.364956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.364870    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365025 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.364985    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.365024    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365131 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.365114    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.365261    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365332 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.365296    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.365392    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.365451 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.365428    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.369118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.369094    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c54de0-openshift-default-dockercfg-k2kn2\""
Nov 01 22:44:41.373939 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.373919    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"]
Nov 01 22:44:41.378052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.378037    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.406789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.406767    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"]
Nov 01 22:44:41.410946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.410930    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.422774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.422751    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"]
Nov 01 22:44:41.441179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.441152    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"]
Nov 01 22:44:41.446357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.446339    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.465765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.464220    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"]
Nov 01 22:44:41.467219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.467191    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.467344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.467240    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.467344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.467269    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.467510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.467456    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.475798    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.475842    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.475880    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.475907    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.475945    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.475972    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.476032    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.476074    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.476142    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.476650    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.476751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.476686    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.477401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.477020    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"]
Nov 01 22:44:41.479219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479191    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.479318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479273    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.479318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479301    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.479448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479329    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rpxp\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.479448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479367    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.479448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479403    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.479448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479427    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmf5\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.479681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.479681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.479681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479571    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.479681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479622    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.479915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479649    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.479915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479716    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.479915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479792    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.479915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479840    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.479915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479873    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.480202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479927    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.480202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.479927    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.480300 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480237    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.480448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480425    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.480567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480548    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.480567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480551    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.480698 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480608    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.480840 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480821    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.481157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480897    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.481157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480962    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c77rw\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.481157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.480990    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.481366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481158    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.481366 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481360    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.481470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481421    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.481716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481695    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.481867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481851    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pntrq\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.481931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481875    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.481931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481897    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwclx\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.481931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481912    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.481931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.481929    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.484438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.484407    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.531981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.517818    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"]
Nov 01 22:44:41.531981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.524943    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"]
Nov 01 22:44:41.561709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.561679    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pntrq\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.565453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.565424    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.583402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583312    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.583402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583369    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwclx\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.583402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583397    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583424    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583455    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583487    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583515    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583550    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzhtf\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583592    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583617    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.583679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583683    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583744    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583774    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583807    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583853    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583885    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.583995    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584035    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584069    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584117    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584157    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584182    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584208    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rpxp\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584241    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584277    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584311    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmf5\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584353    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584398    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584473    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584535    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584600    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584657    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584684    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584717    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584777    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584820    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584851    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584892    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c77rw\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584918    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.584970    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.585020    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.588236    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.588663    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.589026    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.589651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.589532    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.591558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.589816    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.591558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.590050    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.593482 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.593459    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.595188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.595154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.596558    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.596850    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.596867    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"]
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.597029    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.597079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.597391    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.597539    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.597666    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.598080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.597764    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.598598 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.598282    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.598598 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.598354    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.598713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.598655    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.599098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.598965    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.602100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.599191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.602100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.599191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.602100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.599477    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.602100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.601956    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.602358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.602175    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:41.607697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.607358    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.607891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.607869    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.608060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.608044    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.611206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.611184    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.644833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.644797    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwclx\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.645130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.645109    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.653994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.653968    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rpxp\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.654186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.654156    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c77rw\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.654696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.654659    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token\") pod \"e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") " pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.656793 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.656773    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmf5\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.658260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.658049    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:41.661234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.661207    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token\") pod \"e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") " pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.681685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.681629    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token\") pod \"e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") " pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687384    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687451    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687567    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687707    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.687800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687787    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.688202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.687971    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.688202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.688053    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.688202 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.688094    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzhtf\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.689654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.688948    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.689654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.689071    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.689654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.689326    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.689654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.689413    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.690800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.690722    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.690920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.690904    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:41.692855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.692799    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.725635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.725472    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:41.742324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.742279    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.742976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.742945    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzhtf\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.812956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.812898    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:41.880350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.880322    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:41.894566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.894539    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca\") pod \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") "
Nov 01 22:44:41.894915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.894892    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm\" (UniqueName: \"kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm\") pod \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") "
Nov 01 22:44:41.895098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.895070    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca\") pod \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\" (UID: \"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e\") "
Nov 01 22:44:41.896136 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.896112    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca" pod "c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" (UID: "c7ab4e74-c70b-4dd5-823f-d208cf8fb02e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:44:41.896854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.896833    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca" pod "c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" (UID: "c7ab4e74-c70b-4dd5-823f-d208cf8fb02e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:44:41.897880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.897845    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm" pod "c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" (UID: "c7ab4e74-c70b-4dd5-823f-d208cf8fb02e"). InnerVolumeSpecName "kube-api-access-v6grm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:44:41.950109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.949939    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:41.998697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.998580    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6grm\" (UniqueName: \"kubernetes.io/projected/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-kube-api-access-v6grm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:41.998697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.998638    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:41.998697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:41.998659    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:44:42.059474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.058713    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.059474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.058844    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.059474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.058890    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.069827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.069796    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"]
Nov 01 22:44:42.088130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.088100    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"]
Nov 01 22:44:42.089270 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:42.089242    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647cd028_ce48_4e6a_930d_34fc3c8a6342.slice/crio-8a22aead73c12dc7d1e27d1158798675102193d216e6cfd8eaa073015b10a50e WatchSource:0}: Error finding container 8a22aead73c12dc7d1e27d1158798675102193d216e6cfd8eaa073015b10a50e: Status 404 returned error can't find the container with id 8a22aead73c12dc7d1e27d1158798675102193d216e6cfd8eaa073015b10a50e
Nov 01 22:44:42.098219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.097683    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.098219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.097759    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.098219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.097790    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.175069 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.175039    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"]
Nov 01 22:44:42.221598 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.221567    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"]
Nov 01 22:44:42.232420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.232335    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.232699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.232562    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.232699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.232623    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.235636 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.235614    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.235716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.235664    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.235716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.235700    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.307241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.307214    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"]
Nov 01 22:44:42.308530 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:44:42.308494    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6907aea_9a35_4bfc_9cef_d29a0a56c5db.slice/crio-ef85c1ff2a528782a227b9fe463b44978b1e3b10086e11d290de0288de85e45d WatchSource:0}: Error finding container ef85c1ff2a528782a227b9fe463b44978b1e3b10086e11d290de0288de85e45d: Status 404 returned error can't find the container with id ef85c1ff2a528782a227b9fe463b44978b1e3b10086e11d290de0288de85e45d
Nov 01 22:44:42.312484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.312451    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.312562 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.312530    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.312626 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.312569    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.470016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.468685    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"]
Nov 01 22:44:42.478157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.477684    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.478157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.477827    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.478157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.477879    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:44:42.591606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.591569    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" event={"ID":"81aa20da-82c2-42f0-91c1-a547c05f615e","Type":"ContainerStarted","Data":"e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48"}
Nov 01 22:44:42.596159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.596132    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" event={"ID":"81aa20da-82c2-42f0-91c1-a547c05f615e","Type":"ContainerStarted","Data":"76dd1b221963bd9769f5e09c845b0ee9816ccde8b78c901a1d0f0394e892c376"}
Nov 01 22:44:42.705366 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:44:42.705329    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:44:42.706132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.706079    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:42.707331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.707298    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" event={"ID":"14d4724b-596f-4e61-9c42-d187d356ea08","Type":"ContainerStarted","Data":"f968c79073ef885e3659936d700c30c299d036f6d04d7b9a5f56732f7dfb3bd2"}
Nov 01 22:44:42.707401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.707350    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" event={"ID":"14d4724b-596f-4e61-9c42-d187d356ea08","Type":"ContainerStarted","Data":"19641fd2026f863cd09e6dd3cea6cd4d2ef66c2b6597bc040cafb1045dd9b8c6"}
Nov 01 22:44:42.726094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.726027    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:42.726580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.726549    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" event={"ID":"937a1115-7ed7-4a43-bc36-1c2424f7228a","Type":"ContainerStarted","Data":"472c0d610ee8aac98fd57e16001157999e92bf9b4e78943ce79139ea71e237f6"}
Nov 01 22:44:42.735810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.729430    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podStartSLOduration=1.729412926 podStartE2EDuration="1.729412926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.721356772 +0000 UTC m=+1264.039006777" watchObservedRunningTime="2026-11-01 22:44:42.729412926 +0000 UTC m=+1264.047062929"
Nov 01 22:44:42.739506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.738848    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" event={"ID":"c6907aea-9a35-4bfc-9cef-d29a0a56c5db","Type":"ContainerStarted","Data":"3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994"}
Nov 01 22:44:42.739506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.738901    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" event={"ID":"c6907aea-9a35-4bfc-9cef-d29a0a56c5db","Type":"ContainerStarted","Data":"ef85c1ff2a528782a227b9fe463b44978b1e3b10086e11d290de0288de85e45d"}
Nov 01 22:44:42.743279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.743199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" event={"ID":"647cd028-ce48-4e6a-930d-34fc3c8a6342","Type":"ContainerStarted","Data":"44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689"}
Nov 01 22:44:42.743279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.743243    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" event={"ID":"647cd028-ce48-4e6a-930d-34fc3c8a6342","Type":"ContainerStarted","Data":"8a22aead73c12dc7d1e27d1158798675102193d216e6cfd8eaa073015b10a50e"}
Nov 01 22:44:42.745673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.745599    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9" event={"ID":"c7ab4e74-c70b-4dd5-823f-d208cf8fb02e","Type":"ContainerDied","Data":"7d9415b67e0764c2a82c39c438a63fb377dcd186ffb70a4abdd3a538e1fc9fdd"}
Nov 01 22:44:42.745673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.745640    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d9415b67e0764c2a82c39c438a63fb377dcd186ffb70a4abdd3a538e1fc9fdd"
Nov 01 22:44:42.746163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.746144    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"
Nov 01 22:44:42.766141 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.766115    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" event={"ID":"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7","Type":"ContainerStarted","Data":"f1920d96f2e935e4e297a08de0528ecc5cd09ce533fcd34bc604640ee84eb4f9"}
Nov 01 22:44:42.766313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.766297    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" event={"ID":"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7","Type":"ContainerStarted","Data":"d2ab34aa1ae4c23a35b3b2f62b5e69d5b136e0c86b949b144cfb4c4206fc6bbb"}
Nov 01 22:44:42.870906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.870527    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podStartSLOduration=1.870506418 podStartE2EDuration="1.870506418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:42.86799688 +0000 UTC m=+1264.185646917" watchObservedRunningTime="2026-11-01 22:44:42.870506418 +0000 UTC m=+1264.188156437"
Nov 01 22:44:42.883156 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:42.883088    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:43.078749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.077718    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podStartSLOduration=2.077696464 podStartE2EDuration="2.077696464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.073459915 +0000 UTC m=+1264.391109916" watchObservedRunningTime="2026-11-01 22:44:43.077696464 +0000 UTC m=+1264.395346466"
Nov 01 22:44:43.247748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.245325    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podStartSLOduration=2.245304129 podStartE2EDuration="2.245304129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.203366824 +0000 UTC m=+1264.521016827" watchObservedRunningTime="2026-11-01 22:44:43.245304129 +0000 UTC m=+1264.562954130"
Nov 01 22:44:43.368515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.368442    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podStartSLOduration=2.368419475 podStartE2EDuration="2.368419475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:43.334377088 +0000 UTC m=+1264.652027090" watchObservedRunningTime="2026-11-01 22:44:43.368419475 +0000 UTC m=+1264.686069476"
Nov 01 22:44:43.911754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.894396    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:43.917900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.917868    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:43.918117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.898157    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:43.918252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.918230    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:44.027749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.900263    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.165203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.165101    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:44.192335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.913838    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.192618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.192582    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:44.192810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:43.999854    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.192993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.192957    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:44.217753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.029117    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.218238 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.218197    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:44.290893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.290858    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" event={"ID":"937a1115-7ed7-4a43-bc36-1c2424f7228a","Type":"ContainerStarted","Data":"9b5328ecc375649949c031911b11fbf98606f5abf9fa50072fa01633b51ff63f"}
Nov 01 22:44:44.569788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.528008    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.569788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.528437    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:44:44.604189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.604151    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.604485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.604439    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:44.651821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.651782    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"]
Nov 01 22:44:44.667864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.667829    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.668275 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.668227    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:44.690822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.690783    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d212ba5e-fhnv7"]
Nov 01 22:44:44.716753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.691269    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podStartSLOduration=3.691252615 podStartE2EDuration="3.691252615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:44:44.485673279 +0000 UTC m=+1265.803323281" watchObservedRunningTime="2026-11-01 22:44:44.691252615 +0000 UTC m=+1266.008902617"
Nov 01 22:44:44.738827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.691376    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.739120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.739089    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:44.739251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.736847    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.739350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.739334    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:44.845048 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.844960    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e75d00a-9ede-44af-b2f9-74e0d66bf058" path="/var/lib/kubelet/pods/8e75d00a-9ede-44af-b2f9-74e0d66bf058/volumes"
Nov 01 22:44:44.860902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.860861    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.882212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.882167    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.882481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.882457    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:44.895915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.895873    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:44.953267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.950900    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:44.953668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.953632    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:44.953887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.953845    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:44.977751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:44.964932    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.263807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.255845    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": context deadline exceeded"
Nov 01 22:44:45.298995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.295373    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": context deadline exceeded"
Nov 01 22:44:45.298995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.295490    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.298995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.297819    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.306082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.306037    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.422752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.417808    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.534764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.534648    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.558548 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.558497    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:44:45.575892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.575795    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:44:45.576389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.575902    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:44:45.613323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.613282    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.613575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.613541    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:45.727555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.727517    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.735752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.727806    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:45.743759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.730349    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.752842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.752800    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:45.760753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.730391    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.875896 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.875797    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:45.876143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.863801    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.901749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.876544    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:44:45.902779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.882250    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.902986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.902962    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:45.927825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.899702    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.930940 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.930897    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:44:45.952813 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.952776    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:45.953067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.953038    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:45.973856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.973816    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:45.975025 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:45.974990    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:44:46.028246 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.028215    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:44:46.349909 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.342618    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:46.377928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.368634    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:46.402076 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.402048    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:44:46.426187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.421858    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.455109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.455082    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:44:46.563249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.563208    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:46.887990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:46.887942    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.015753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.004552    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.028762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.022830    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.049582 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.049543    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.287868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.287807    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.216237    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.216657    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.216876    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.204496    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.058759    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.292708    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.058812    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.292791    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.066872    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.292839    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.152233    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.297756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.292886    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:47.552832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.317452    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.552832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.317515    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:47.552832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.403300    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.552832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.451382    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.624763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.605035    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.624763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.605112    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:47.780233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.780184    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.780557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.780508    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:47.805313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.805215    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:47.883893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.883826    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:47.884240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.884209    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:47.926755 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.891067    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.002792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.998901    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:48.010792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.004847    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.031327 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.911764    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.031854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:47.911692    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.032114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.032084    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:48.032373 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.030933    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.061797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.058640    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:48.199906 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.096814    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.309032 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.308994    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.317857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.313809    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:44:48.345888 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.345831    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.447753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.438929    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.481752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.461165    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.645846 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:44:48.625915    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.448s"
Nov 01 22:44:48.646475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.646443    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:44:48.646697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.646681    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:44:48.646893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.646876    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:44:48.695427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.695391    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.725752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.705806    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:48.725752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.703122    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.725752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.705899    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:48.725752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.718782    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.725752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.718928    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:48.739925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.739881    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:48.740193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.740165    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:48.817756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.811041    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:48.994843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:48.994788    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.011827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.011767    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:49.022807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.015895    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.046897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.046860    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:49.051760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.048150    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.051760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.049784    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.225753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.200414    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.352969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.351311    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.352969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.352107    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.352969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.352375    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.381672 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.381642    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:44:49.501751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.464913    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:49.501751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.482081    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.781859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.781823    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.782126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.782095    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:49.782307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.782289    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.782415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.782398    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:49.782589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.782571    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.782721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.782703    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:49.788843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.788817    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.789009 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.788987    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:49.862756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.811392    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:49.877866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.877832    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"]
Nov 01 22:44:49.906472 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.906440    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:49.938873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:49.938826    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:50.046820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.034856    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.046820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.034928    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:50.073752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.048922    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.102638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.050255    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.103455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.050620    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": context deadline exceeded"
Nov 01 22:44:50.284324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.284289    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b0f084d-klgl9"]
Nov 01 22:44:50.366801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.361805    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.393670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.393627    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.394201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.394169    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.493815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.493774    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.496421 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.496390    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.508298 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.503878    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.608540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.608505    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.608836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.608804    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:50.663338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.663253    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.663616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.663590    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:50.704829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.697306    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.704829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.697364    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:50.704829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.703808    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:50.727201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.727164    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.730750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.727452    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:50.890512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.890470    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:44:50.893827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.893796    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.899834 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.899791    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:50.900221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.900189    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d2b247-7e69-4ba1-b043-496b56110ab5" path="/var/lib/kubelet/pods/44d2b247-7e69-4ba1-b043-496b56110ab5/volumes"
Nov 01 22:44:50.951205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.951170    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:50.951544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:50.951513    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:51.107749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.104973    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.107749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.104916    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.200873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.200820    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.442803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.396446    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.442803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.405123    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.442803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.405166    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.498857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.498806    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.508212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.508172    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 01 22:44:51.634873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.634836    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.674864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.674818    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:51.763879 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.758841    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.764191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.764157    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:51.775059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.760896    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.775247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.775223    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:51.775369 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.761591    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.775462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.775447    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:51.881892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.881857    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:51.912841 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.912807    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": context deadline exceeded"
Nov 01 22:44:51.913070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.912801    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:51.913273 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.913249    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:51.998397 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:51.998361    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.019902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.019806    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:52.105869 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.105817    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.109762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.108838    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.109762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.109171    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/health\": context deadline exceeded"
Nov 01 22:44:52.205971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.205910    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.206720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.206697    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:44:52.449754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.448964    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.449754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.449336    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.449754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.449645    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.509756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.507300    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:52.509756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.508543    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:52.623751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.616945    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.623751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.616999    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:52.677358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.676875    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.677358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.676935    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:52.697679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.697645    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.697930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.697904    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:52.743802 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.735911    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:52.743802 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.735969    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:52.929259 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:52.921633    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.131633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.121776    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.131633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.121841    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:53.135067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.135035    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.135278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.135248    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:53.199785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.192977    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.253920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.225197    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.253920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.230865    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.559795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.530669    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.576876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.576831    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.639310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.638829    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.699773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.690542    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded"
Nov 01 22:44:53.741763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.735214    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.763880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.763834    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:53.764260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.764235    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.764413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.764392    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:53.764608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.764581    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:53.765035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.765007    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 01 22:44:53.783452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.771478    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.783702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.783672    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:53.791756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.784425    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:53.876148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.876040    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:54.079367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.939950    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.079672 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.079641    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:54.079880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:53.972680    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.080399 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.000705    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.149121 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.149012    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:54.205806 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:44:54.153083    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.319s"
Nov 01 22:44:54.230750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.198535    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.233932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.233882    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.234211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.234184    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:54.568404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.568357    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:44:54.614905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.614859    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.660752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.627853    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.660752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.627926    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:54.660752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.653999    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.664555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.664516    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.664832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.664781    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:54.726766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.697864    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.726766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.697925    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:54.766752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.741710    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.767067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.767027    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:54.767286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.767268    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:54.788293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.767685    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:44:54.911644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.911548    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:54.917034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:54.916995    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:55.002008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.001970    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.002277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.002249    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:55.035348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.035299    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.080443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.080403    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.128752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.101017    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.406751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.377554    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.511752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.388567    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:55.511752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.388638    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:44:55.511752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.408551    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:44:55.576750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.571323    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.670809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.638035    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.680760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.680355    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:55.680760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.679785    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.705075    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.705135    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.705393    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.705422    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.712668    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.727316    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.727372    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:55.794771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.780578    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.849151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.849106    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:55.902280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.902243    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.902558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.902528    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:55.954750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.950975    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:55.955035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:55.954994    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:56.117219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.116832    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:56.234193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.233772    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"]
Nov 01 22:44:56.300753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.290496    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-886c94c7-xkt4f"]
Nov 01 22:44:56.484693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.483830    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:56.490234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.488835    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:44:56.490234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.489161    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:56.587830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.571671    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:56.678748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.678643    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:56.678748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.678706    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:56.732748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.710186    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:56.732748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.712062    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:56.732748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.712111    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:56.801638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.797688    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:56.801638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.799406    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:44:56.801638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:56.800168    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:57.082670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.082565    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:57.096895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.096841    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:57.121687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.121655    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.121946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.121916    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:57.123565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.123541    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.123761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.123710    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:57.128837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.128811    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.129014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.128991    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Nov 01 22:44:57.134989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.134965    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.135167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.135144    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:57.135415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.135391    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:57.184613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.184570    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be2df68a-8101-491a-901e-6191f12ec192" path="/var/lib/kubelet/pods/be2df68a-8101-491a-901e-6191f12ec192/volumes"
Nov 01 22:44:57.185601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.185571    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"]
Nov 01 22:44:57.192382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.192349    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" containerName="revoke-keys"
Nov 01 22:44:57.192525 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.192509    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" containerName="revoke-keys"
Nov 01 22:44:57.193650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.193619    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" containerName="revoke-keys"
Nov 01 22:44:57.288752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.206918    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": context deadline exceeded"
Nov 01 22:44:57.342053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.341984    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:57.342278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.342265    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:57.359749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.353994    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:57.359749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.354053    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:57.359749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.343862    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.398788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.369917    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"]
Nov 01 22:44:57.486099 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.484817    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:57.497024 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.496985    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:44:57.497647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.497600    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:44:57.564078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.563921    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.563996    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564158    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564398 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564370    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564461 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564435    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk7h9\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564485    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564691    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.564842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.564825    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.591759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.589119    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:57.607234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.607064    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.607234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.607125    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Nov 01 22:44:57.685749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.665094    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.685749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.665164    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused"
Nov 01 22:44:57.685749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.667082    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.685749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.667144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.694200    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.694529    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.694572    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.704968    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705092    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705156    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk7h9\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705205    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705244    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705347    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705427    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.705935    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.706641    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.706985    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.729756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.708352    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.804849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.804815    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.832690 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.805088    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:57.848751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.805129    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:44:57.861217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.815887    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:57.891759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.891053    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.891759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.891105    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": dial tcp 10.133.0.124:15021: connect: connection refused"
Nov 01 22:44:57.914795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.913583    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:44:57.991758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.986829    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused" start-of-body=
Nov 01 22:44:57.991758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:57.986887    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.120:15021/healthz/ready\": dial tcp 10.133.0.120:15021: connect: connection refused"
Nov 01 22:44:58.036657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.036625    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk7h9\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:58.062196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.062135    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:58.067431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.067407    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:44:58.073752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.071548    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:44:58.090748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.087797    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:58.175057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.167067    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:58.175057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.169036    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:58.426804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.422937    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:44:58.457857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.456772    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:44:58.522592 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.522550    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:58.565100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.565068    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:58.571330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.571304    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:44:58.866759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:58.862561    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.117826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.107605    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.118451 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.118416    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.182746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.175459    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.202746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.175813    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.456114 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.407993    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:44:59.470762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.461947    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.520871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.519414    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:44:59.520871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.519478    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:44:59.522610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.522590    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:44:59.601960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.598855    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.664943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.657653    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:59.754081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.753994    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:44:59.772751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.762740    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.798006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.776340    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:59.808748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.798451    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:59.808748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.800098    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:44:59.808748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.802532    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:44:59.808748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.805559    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:44:59.812940 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.812919    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 22:44:59.821496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.820273    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:44:59.884324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.884285    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:44:59.885700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.885636    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:44:59.889463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.889441    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:59.898031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.897209    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:44:59.898387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.898366    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 22:44:59.905958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:44:59.905907    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:45:00.214335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.214304    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"]
Nov 01 22:45:00.219108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.219088    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.228368 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.228343    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"]
Nov 01 22:45:00.230851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.230825    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdpj2\" (UniqueName: \"kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.230988 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.230894    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.233409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.233391    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.272468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.266175    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"]
Nov 01 22:45:00.275719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.273496    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.280751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.278262    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:00.281462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.281008    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"]
Nov 01 22:45:00.293185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.293164    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.323848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.322790    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"]
Nov 01 22:45:00.331070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.330307    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"]
Nov 01 22:45:00.332188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332155    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbgp9\" (UniqueName: \"kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.332293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332215    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.332293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332276    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc6bk\" (UniqueName: \"kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.332418 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332341    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdpj2\" (UniqueName: \"kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.332418 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332390    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.332549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332431    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.332549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332521    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrqp\" (UniqueName: \"kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.332656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.332605    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.341427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.333664    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.344826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.344802    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"]
Nov 01 22:45:00.362120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.361476    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:00.362120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.361561    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:00.362120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.361607    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:00.387652 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.387622    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdpj2\" (UniqueName: \"kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2\") pod \"maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.388796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.388769    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"]
Nov 01 22:45:00.399774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.396245    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"]
Nov 01 22:45:00.434058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434029    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc6bk\" (UniqueName: \"kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.434200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434101    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.434200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434186    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrqp\" (UniqueName: \"kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.434322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434280    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.434423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434404    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbgp9\" (UniqueName: \"kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.434495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434454    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.434934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.434908    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.435331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.435307    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.435411 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.435314    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.491997 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.491965    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc6bk\" (UniqueName: \"kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk\") pod \"maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.494970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.494923    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbgp9\" (UniqueName: \"kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9\") pod \"maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.495615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.495573    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrqp\" (UniqueName: \"kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.541748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.538383    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:00.549356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.549138    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:00.601000 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.600941    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:00.634577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.628280    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:00.831547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.831515    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" event={"ID":"113370e8-9731-4fb5-b6b9-474cc477c413","Type":"ContainerStarted","Data":"8b27f3a2b125d09b987a39fd8c8790789fdb7da4adbf391b4c84d3ccce56bee9"}
Nov 01 22:45:00.831842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:00.831817    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" event={"ID":"113370e8-9731-4fb5-b6b9-474cc477c413","Type":"ContainerStarted","Data":"11a803db9d250fd8bb0b86636b1a084b85915abcab8e89a6be4d486c7beadd9e"}
Nov 01 22:45:01.050854 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.050042    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"]
Nov 01 22:45:01.062250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.062225    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"]
Nov 01 22:45:01.066403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.066380    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"]
Nov 01 22:45:01.073210 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.073185    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00f5df69-dzszf"]
Nov 01 22:45:01.073377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.073363    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:45:01.076011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.075985    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:01.076119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.076041    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:01.099517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.099481    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"]
Nov 01 22:45:01.178280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.178243    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"]
Nov 01 22:45:01.299341 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:01.299230    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8d5fb4_97c9_4b20_bfc9_849fe95aae33.slice/crio-2934e0b191fdf67461e3bad4fb30e4041de004a73c3d54c55f48b276cc9a80a7 WatchSource:0}: Error finding container 2934e0b191fdf67461e3bad4fb30e4041de004a73c3d54c55f48b276cc9a80a7: Status 404 returned error can't find the container with id 2934e0b191fdf67461e3bad4fb30e4041de004a73c3d54c55f48b276cc9a80a7
Nov 01 22:45:01.380750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.379873    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:01.651718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.648799    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podStartSLOduration=5.648785294 podStartE2EDuration="5.648785294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:01.610052647 +0000 UTC m=+1282.927702651" watchObservedRunningTime="2026-11-01 22:45:01.648785294 +0000 UTC m=+1282.966435294"
Nov 01 22:45:01.652106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.652026    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"]
Nov 01 22:45:01.657418 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.657400    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.710119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.710092    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:01.712597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.712569    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"]
Nov 01 22:45:01.715777 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.715711    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.715973 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.715952    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.716241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.716221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbw2x\" (UniqueName: \"kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.817995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.817957    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbw2x\" (UniqueName: \"kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.818157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.818142    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.818206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.818166    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.818798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.818781    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.820358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.820334    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.840463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.840429    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerStarted","Data":"c50f351b9d2e813406a9acd93c74ccc4c52fb3d4c4443f042766454e967649eb"}
Nov 01 22:45:01.840617 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.840489    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerStarted","Data":"b9ec975823d55731d0b20a2c5854c1d38a3863b695053fb9bebb35f744e3c342"}
Nov 01 22:45:01.842170 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.842143    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerStarted","Data":"c80a27206dab5c4e6e27b9d07f012afa98f30c1cd25fa628402a6c3637529e55"}
Nov 01 22:45:01.842295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.842188    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerStarted","Data":"2934e0b191fdf67461e3bad4fb30e4041de004a73c3d54c55f48b276cc9a80a7"}
Nov 01 22:45:01.843850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.843826    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerStarted","Data":"a9875f2ea364aa86c7904e473e67a64efdde706eed246f82b2f8a55faf1e302f"}
Nov 01 22:45:01.843969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.843942    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerStarted","Data":"e3ba6f0bec0ed00e19df484098fc7a50ebfec28932c465f47ee4d26804235ff7"}
Nov 01 22:45:01.845473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.845451    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerStarted","Data":"0ce5444bad457c49157441d6796750b53662d9dfca9f95cea6d3325f04a2dddd"}
Nov 01 22:45:01.845575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.845492    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerStarted","Data":"70be786d93fb5695c447a42f39fa95e34ea335601b7c0501493cb2cb79d6681a"}
Nov 01 22:45:01.892541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.892460    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbw2x\" (UniqueName: \"kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x\") pod \"maas-api-revoke-sub-57144b1c-8rqbj\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:01.992187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:01.992123    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" podStartSLOduration=1.992108746 podStartE2EDuration="1.992108746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:01.989009457 +0000 UTC m=+1283.306659483" watchObservedRunningTime="2026-11-01 22:45:01.992108746 +0000 UTC m=+1283.309758783"
Nov 01 22:45:02.005076 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.005047    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:02.082132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.074823    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:02.082132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.074877    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:02.117060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.116997    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" podStartSLOduration=2.116965006 podStartE2EDuration="2.116965006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.112989302 +0000 UTC m=+1283.430639305" watchObservedRunningTime="2026-11-01 22:45:02.116965006 +0000 UTC m=+1283.434615008"
Nov 01 22:45:02.287656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.281640    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" podStartSLOduration=2.281618957 podStartE2EDuration="2.281618957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.257879818 +0000 UTC m=+1283.575529819" watchObservedRunningTime="2026-11-01 22:45:02.281618957 +0000 UTC m=+1283.599268959"
Nov 01 22:45:02.396894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.396848    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" podStartSLOduration=2.396832973 podStartE2EDuration="2.396832973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:02.384720158 +0000 UTC m=+1283.702370160" watchObservedRunningTime="2026-11-01 22:45:02.396832973 +0000 UTC m=+1283.714483025"
Nov 01 22:45:02.411119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.411092    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"]
Nov 01 22:45:02.820873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.820840    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aaab0b9-11e2-4f72-9353-0854513d01f8" path="/var/lib/kubelet/pods/2aaab0b9-11e2-4f72-9353-0854513d01f8/volumes"
Nov 01 22:45:02.877337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.877302    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerID="c50f351b9d2e813406a9acd93c74ccc4c52fb3d4c4443f042766454e967649eb" exitCode=60
Nov 01 22:45:02.882274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.882249    4531 scope.go:122] "RemoveContainer" containerID="c50f351b9d2e813406a9acd93c74ccc4c52fb3d4c4443f042766454e967649eb"
Nov 01 22:45:02.887011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.886967    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerDied","Data":"c50f351b9d2e813406a9acd93c74ccc4c52fb3d4c4443f042766454e967649eb"}
Nov 01 22:45:02.905714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.905675    4531 generic.go:356] "Generic (PLEG): container finished" podID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerID="c80a27206dab5c4e6e27b9d07f012afa98f30c1cd25fa628402a6c3637529e55" exitCode=60
Nov 01 22:45:02.905848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.905798    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerDied","Data":"c80a27206dab5c4e6e27b9d07f012afa98f30c1cd25fa628402a6c3637529e55"}
Nov 01 22:45:02.906277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.906255    4531 scope.go:122] "RemoveContainer" containerID="c80a27206dab5c4e6e27b9d07f012afa98f30c1cd25fa628402a6c3637529e55"
Nov 01 22:45:02.908478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.908452    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj" event={"ID":"a0af721f-fd65-4794-9c71-29b6cb7aa627","Type":"ContainerStarted","Data":"ef6e85037a31d8cafc6acf47db5f85309b88f71aef3f0be39644c33aea3af02f"}
Nov 01 22:45:02.911499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.910640    4531 generic.go:356] "Generic (PLEG): container finished" podID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerID="a9875f2ea364aa86c7904e473e67a64efdde706eed246f82b2f8a55faf1e302f" exitCode=60
Nov 01 22:45:02.911499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.910769    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerDied","Data":"a9875f2ea364aa86c7904e473e67a64efdde706eed246f82b2f8a55faf1e302f"}
Nov 01 22:45:02.911499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.911140    4531 scope.go:122] "RemoveContainer" containerID="a9875f2ea364aa86c7904e473e67a64efdde706eed246f82b2f8a55faf1e302f"
Nov 01 22:45:02.917175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.916980    4531 generic.go:356] "Generic (PLEG): container finished" podID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerID="0ce5444bad457c49157441d6796750b53662d9dfca9f95cea6d3325f04a2dddd" exitCode=60
Nov 01 22:45:02.918292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.918277    4531 scope.go:122] "RemoveContainer" containerID="0ce5444bad457c49157441d6796750b53662d9dfca9f95cea6d3325f04a2dddd"
Nov 01 22:45:02.918689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:02.918671    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerDied","Data":"0ce5444bad457c49157441d6796750b53662d9dfca9f95cea6d3325f04a2dddd"}
Nov 01 22:45:03.092954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.092831    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:03.092954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.092893    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:03.929050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.928964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerStarted","Data":"c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780"}
Nov 01 22:45:03.930781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.930719    4531 generic.go:356] "Generic (PLEG): container finished" podID="a0af721f-fd65-4794-9c71-29b6cb7aa627" containerID="c5ef009573b33d2de4ebeb5a9f26e4626a0a40f40de2c3dd71080eb1c0dd9329" exitCode=0
Nov 01 22:45:03.930913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.930793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj" event={"ID":"a0af721f-fd65-4794-9c71-29b6cb7aa627","Type":"ContainerDied","Data":"c5ef009573b33d2de4ebeb5a9f26e4626a0a40f40de2c3dd71080eb1c0dd9329"}
Nov 01 22:45:03.932887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.932863    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerStarted","Data":"7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0"}
Nov 01 22:45:03.934907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.934882    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerStarted","Data":"57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519"}
Nov 01 22:45:03.936931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:03.936908    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerStarted","Data":"83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e"}
Nov 01 22:45:04.077762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.072691    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:04.077762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.072774    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:04.950013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.949976    4531 generic.go:356] "Generic (PLEG): container finished" podID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerID="c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780" exitCode=60
Nov 01 22:45:04.950516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.950064    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerDied","Data":"c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780"}
Nov 01 22:45:04.950516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.950125    4531 scope.go:122] "RemoveContainer" containerID="c80a27206dab5c4e6e27b9d07f012afa98f30c1cd25fa628402a6c3637529e55"
Nov 01 22:45:04.950516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.950480    4531 scope.go:122] "RemoveContainer" containerID="c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780"
Nov 01 22:45:04.950988 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:04.950924    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6_odh-ai-gateway-infra(1b8d5fb4-97c9-4b20-bfc9-849fe95aae33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33"
Nov 01 22:45:04.952118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.952092    4531 generic.go:356] "Generic (PLEG): container finished" podID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerID="7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0" exitCode=60
Nov 01 22:45:04.952251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.952227    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerDied","Data":"7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0"}
Nov 01 22:45:04.952428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.952415    4531 scope.go:122] "RemoveContainer" containerID="7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0"
Nov 01 22:45:04.952766 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:04.952741    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5_odh-ai-gateway-infra(1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5"
Nov 01 22:45:04.954335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.954316    4531 generic.go:356] "Generic (PLEG): container finished" podID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerID="57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519" exitCode=60
Nov 01 22:45:04.954715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.954448    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerDied","Data":"57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519"}
Nov 01 22:45:04.954715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.954643    4531 scope.go:122] "RemoveContainer" containerID="57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519"
Nov 01 22:45:04.955341 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:04.954957    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64_odh-ai-gateway-infra(f895a121-9216-45e9-a694-56e7c3d4ac9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a"
Nov 01 22:45:04.956682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.956658    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerID="83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e" exitCode=60
Nov 01 22:45:04.956769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.956689    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerDied","Data":"83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e"}
Nov 01 22:45:04.957006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.956991    4531 scope.go:122] "RemoveContainer" containerID="83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e"
Nov 01 22:45:04.957265 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:04.957249    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7_odh-ai-gateway-infra(ea053b55-4cca-4af7-a05d-7aff6c5cdf93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93"
Nov 01 22:45:04.992788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:04.992762    4531 scope.go:122] "RemoveContainer" containerID="a9875f2ea364aa86c7904e473e67a64efdde706eed246f82b2f8a55faf1e302f"
Nov 01 22:45:05.013152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.013117    4531 scope.go:122] "RemoveContainer" containerID="0ce5444bad457c49157441d6796750b53662d9dfca9f95cea6d3325f04a2dddd"
Nov 01 22:45:05.029021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.029003    4531 scope.go:122] "RemoveContainer" containerID="c50f351b9d2e813406a9acd93c74ccc4c52fb3d4c4443f042766454e967649eb"
Nov 01 22:45:05.111635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.096090    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:05.111635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.096178    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:05.235852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.235829    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:05.412558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.411068    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca\") pod \"a0af721f-fd65-4794-9c71-29b6cb7aa627\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") "
Nov 01 22:45:05.412558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.411298    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x\" (UniqueName: \"kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x\") pod \"a0af721f-fd65-4794-9c71-29b6cb7aa627\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") "
Nov 01 22:45:05.412558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.411427    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca\") pod \"a0af721f-fd65-4794-9c71-29b6cb7aa627\" (UID: \"a0af721f-fd65-4794-9c71-29b6cb7aa627\") "
Nov 01 22:45:05.413315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.412997    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca" pod "a0af721f-fd65-4794-9c71-29b6cb7aa627" (UID: "a0af721f-fd65-4794-9c71-29b6cb7aa627"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:05.419278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.419244    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca" pod "a0af721f-fd65-4794-9c71-29b6cb7aa627" (UID: "a0af721f-fd65-4794-9c71-29b6cb7aa627"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:05.421055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.421028    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x" pod "a0af721f-fd65-4794-9c71-29b6cb7aa627" (UID: "a0af721f-fd65-4794-9c71-29b6cb7aa627"). InnerVolumeSpecName "kube-api-access-qbw2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:05.514312 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.514241    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:05.514312 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.514271    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0af721f-fd65-4794-9c71-29b6cb7aa627-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:05.514312 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.514281    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbw2x\" (UniqueName: \"kubernetes.io/projected/a0af721f-fd65-4794-9c71-29b6cb7aa627-kube-api-access-qbw2x\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:05.719008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.718970    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:05.991338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.991298    4531 scope.go:122] "RemoveContainer" containerID="57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519"
Nov 01 22:45:05.991802 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:05.991696    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64_odh-ai-gateway-infra(f895a121-9216-45e9-a694-56e7c3d4ac9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a"
Nov 01 22:45:05.993168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.993143    4531 scope.go:122] "RemoveContainer" containerID="83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e"
Nov 01 22:45:05.993477 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:05.993454    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7_odh-ai-gateway-infra(ea053b55-4cca-4af7-a05d-7aff6c5cdf93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93"
Nov 01 22:45:05.995019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.994988    4531 scope.go:122] "RemoveContainer" containerID="c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780"
Nov 01 22:45:05.995296 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:05.995274    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6_odh-ai-gateway-infra(1b8d5fb4-97c9-4b20-bfc9-849fe95aae33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33"
Nov 01 22:45:05.996203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.996181    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"
Nov 01 22:45:05.996319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.996209    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj" event={"ID":"a0af721f-fd65-4794-9c71-29b6cb7aa627","Type":"ContainerDied","Data":"ef6e85037a31d8cafc6acf47db5f85309b88f71aef3f0be39644c33aea3af02f"}
Nov 01 22:45:05.996319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.996252    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef6e85037a31d8cafc6acf47db5f85309b88f71aef3f0be39644c33aea3af02f"
Nov 01 22:45:05.998215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:05.998198    4531 scope.go:122] "RemoveContainer" containerID="7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0"
Nov 01 22:45:05.998415 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:05.998398    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5_odh-ai-gateway-infra(1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5"
Nov 01 22:45:06.084795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:06.083330    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:06.084795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:06.083376    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:06.683751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:06.680295    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:07.071848 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.071807    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Nov 01 22:45:07.072228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.071876    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Nov 01 22:45:07.484058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.484019    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"]
Nov 01 22:45:07.485215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.485191    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a0af721f-fd65-4794-9c71-29b6cb7aa627" containerName="revoke-keys"
Nov 01 22:45:07.485294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.485219    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0af721f-fd65-4794-9c71-29b6cb7aa627" containerName="revoke-keys"
Nov 01 22:45:07.485473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.485456    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0af721f-fd65-4794-9c71-29b6cb7aa627" containerName="revoke-keys"
Nov 01 22:45:07.490963 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.490942    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.542455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.542427    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:07.547090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.547063    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"]
Nov 01 22:45:07.652098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.652064    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtbpw\" (UniqueName: \"kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.652301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.652192    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.652301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.652237    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.753566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.753496    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtbpw\" (UniqueName: \"kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.753701 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.753584    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.753701 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.753630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.754286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.754264    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.756109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.756086    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:07.808508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:07.808480    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtbpw\" (UniqueName: \"kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw\") pod \"maas-api-revoke-sub-c54f5e5e-rjmzf\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:08.107710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:08.107624    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:08.433561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:08.433435    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:45:08.434377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:08.434317    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:45:08.434972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:08.434950    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:45:08.531740 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:08.531696    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"]
Nov 01 22:45:08.534474 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:08.534444    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea3d6ba_85c4_443c_ad1f_d25de767e381.slice/crio-493a88e0d0b368426230f92e2dfbc56baed1b0f8d82e9d4ccab9979c7bc0496f WatchSource:0}: Error finding container 493a88e0d0b368426230f92e2dfbc56baed1b0f8d82e9d4ccab9979c7bc0496f: Status 404 returned error can't find the container with id 493a88e0d0b368426230f92e2dfbc56baed1b0f8d82e9d4ccab9979c7bc0496f
Nov 01 22:45:09.035343 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:09.035304    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf" event={"ID":"dea3d6ba-85c4-443c-ad1f-d25de767e381","Type":"ContainerStarted","Data":"2a0dfea68aedc8c78af60e5d9a75cecae7d52383c332620f854bb4270acdffde"}
Nov 01 22:45:09.035530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:09.035373    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf" event={"ID":"dea3d6ba-85c4-443c-ad1f-d25de767e381","Type":"ContainerStarted","Data":"493a88e0d0b368426230f92e2dfbc56baed1b0f8d82e9d4ccab9979c7bc0496f"}
Nov 01 22:45:09.150486 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:09.150400    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf" podStartSLOduration=2.150386958 podStartE2EDuration="2.150386958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:09.150266776 +0000 UTC m=+1290.467916781" watchObservedRunningTime="2026-11-01 22:45:09.150386958 +0000 UTC m=+1290.468037147"
Nov 01 22:45:10.046542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:10.046509    4531 generic.go:356] "Generic (PLEG): container finished" podID="dea3d6ba-85c4-443c-ad1f-d25de767e381" containerID="2a0dfea68aedc8c78af60e5d9a75cecae7d52383c332620f854bb4270acdffde" exitCode=0
Nov 01 22:45:10.046712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:10.046597    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf" event={"ID":"dea3d6ba-85c4-443c-ad1f-d25de767e381","Type":"ContainerDied","Data":"2a0dfea68aedc8c78af60e5d9a75cecae7d52383c332620f854bb4270acdffde"}
Nov 01 22:45:11.265465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.265442    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:11.302173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.302148    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca\") pod \"dea3d6ba-85c4-443c-ad1f-d25de767e381\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") "
Nov 01 22:45:11.302306 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.302192    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw\" (UniqueName: \"kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw\") pod \"dea3d6ba-85c4-443c-ad1f-d25de767e381\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") "
Nov 01 22:45:11.302358 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.302319    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca\") pod \"dea3d6ba-85c4-443c-ad1f-d25de767e381\" (UID: \"dea3d6ba-85c4-443c-ad1f-d25de767e381\") "
Nov 01 22:45:11.302692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.302667    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca" pod "dea3d6ba-85c4-443c-ad1f-d25de767e381" (UID: "dea3d6ba-85c4-443c-ad1f-d25de767e381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:11.302881 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.302864    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:11.304317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.304291    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca" pod "dea3d6ba-85c4-443c-ad1f-d25de767e381" (UID: "dea3d6ba-85c4-443c-ad1f-d25de767e381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:11.304510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.304488    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw" pod "dea3d6ba-85c4-443c-ad1f-d25de767e381" (UID: "dea3d6ba-85c4-443c-ad1f-d25de767e381"). InnerVolumeSpecName "kube-api-access-qtbpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:11.404624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.404588    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea3d6ba-85c4-443c-ad1f-d25de767e381-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:11.404624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.404625    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtbpw\" (UniqueName: \"kubernetes.io/projected/dea3d6ba-85c4-443c-ad1f-d25de767e381-kube-api-access-qtbpw\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:11.509084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.509022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:11.740686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.740651    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"]
Nov 01 22:45:11.741416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.741391    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dea3d6ba-85c4-443c-ad1f-d25de767e381" containerName="revoke-keys"
Nov 01 22:45:11.741416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.741407    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea3d6ba-85c4-443c-ad1f-d25de767e381" containerName="revoke-keys"
Nov 01 22:45:11.741621 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.741530    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea3d6ba-85c4-443c-ad1f-d25de767e381" containerName="revoke-keys"
Nov 01 22:45:11.745435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.745416    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.811506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.811472    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.814866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.814832    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.814944    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.814986    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.815024    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.815047    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4nqc\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.815079    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.815147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.815856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.815228    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.817256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.817229    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"]
Nov 01 22:45:11.917115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917087    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917131    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917161    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4nqc\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917313    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917420    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917550    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.917769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.917648    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.918102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.918079    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.918201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.918105    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.918276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.918233    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.918276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.918262    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.919772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.919750    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.920213 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.920195    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.971824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.971801    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:11.971938 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:11.971920    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4nqc\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc\") pod \"e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") " pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:12.065605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.065536    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"
Nov 01 22:45:12.065605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.065572    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf" event={"ID":"dea3d6ba-85c4-443c-ad1f-d25de767e381","Type":"ContainerDied","Data":"493a88e0d0b368426230f92e2dfbc56baed1b0f8d82e9d4ccab9979c7bc0496f"}
Nov 01 22:45:12.065782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.065624    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493a88e0d0b368426230f92e2dfbc56baed1b0f8d82e9d4ccab9979c7bc0496f"
Nov 01 22:45:12.067551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.067535    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:12.453147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.453113    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"]
Nov 01 22:45:12.461520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.461261    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.511931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.511903    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:12.519702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.519675    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"]
Nov 01 22:45:12.525998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.525964    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696t5\" (UniqueName: \"kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.526152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.526126    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.526438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.526409    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.539760 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:12.539714    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:45:12.586565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.586543    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"]
Nov 01 22:45:12.589542 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:12.589517    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dbb1c78_2cc1_474a_9734_420b9be02178.slice/crio-b2f7512ba43f85baf8d49dbde6db34ec85a5d7c9adb4a9a118fb150aba7191d2 WatchSource:0}: Error finding container b2f7512ba43f85baf8d49dbde6db34ec85a5d7c9adb4a9a118fb150aba7191d2: Status 404 returned error can't find the container with id b2f7512ba43f85baf8d49dbde6db34ec85a5d7c9adb4a9a118fb150aba7191d2
Nov 01 22:45:12.592170 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.592143    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:12.592252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.592201    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:12.592252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.592230    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:12.628204 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.628179    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.628290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.628267    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696t5\" (UniqueName: \"kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.628362 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.628346    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.628883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.628865    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.630613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.630594    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.685063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.685038    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-696t5\" (UniqueName: \"kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5\") pod \"maas-api-revoke-sub-24c9ef6c-lm82c\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:12.783931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:12.783758    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:13.092805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:13.092678    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" event={"ID":"1dbb1c78-2cc1-474a-9734-420b9be02178","Type":"ContainerStarted","Data":"45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c"}
Nov 01 22:45:13.092977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:13.092806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" event={"ID":"1dbb1c78-2cc1-474a-9734-420b9be02178","Type":"ContainerStarted","Data":"b2f7512ba43f85baf8d49dbde6db34ec85a5d7c9adb4a9a118fb150aba7191d2"}
Nov 01 22:45:13.099154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:13.099123    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"]
Nov 01 22:45:13.105605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:13.105579    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a9f237a-7m6hm"]
Nov 01 22:45:13.184663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:13.184633    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"]
Nov 01 22:45:13.187835 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:13.187808    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad9c4429_dac1_4fae_9132_a815c3e3718f.slice/crio-91108dd026968136b8b12ca15c74fd238a3d7024f8474ab1803c5db0fe07f9fb WatchSource:0}: Error finding container 91108dd026968136b8b12ca15c74fd238a3d7024f8474ab1803c5db0fe07f9fb: Status 404 returned error can't find the container with id 91108dd026968136b8b12ca15c74fd238a3d7024f8474ab1803c5db0fe07f9fb
Nov 01 22:45:13.218946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:13.218892    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podStartSLOduration=2.218874608 podStartE2EDuration="2.218874608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:13.21128405 +0000 UTC m=+1294.528934050" watchObservedRunningTime="2026-11-01 22:45:13.218874608 +0000 UTC m=+1294.536524610"
Nov 01 22:45:14.083104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.083069    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:14.090773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.087823    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 01 22:45:14.090773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.087877    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 01 22:45:14.110970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.110931    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c" event={"ID":"ad9c4429-dac1-4fae-9132-a815c3e3718f","Type":"ContainerStarted","Data":"41f2abbaa359b2f13f8649289b0d629df9ad444ffd4979d9ea4044970d60b4b1"}
Nov 01 22:45:14.111135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.111010    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c" event={"ID":"ad9c4429-dac1-4fae-9132-a815c3e3718f","Type":"ContainerStarted","Data":"91108dd026968136b8b12ca15c74fd238a3d7024f8474ab1803c5db0fe07f9fb"}
Nov 01 22:45:14.231403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.231346    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c" podStartSLOduration=2.23133229 podStartE2EDuration="2.23133229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:14.228831803 +0000 UTC m=+1295.546481804" watchObservedRunningTime="2026-11-01 22:45:14.23133229 +0000 UTC m=+1295.548982290"
Nov 01 22:45:14.810242 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.810209    4531 scope.go:122] "RemoveContainer" containerID="57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519"
Nov 01 22:45:14.811032 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.810944    4531 scope.go:122] "RemoveContainer" containerID="7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0"
Nov 01 22:45:14.820234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:14.820208    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="411cba67-a78b-4479-98e8-588fc9a4e2ce" path="/var/lib/kubelet/pods/411cba67-a78b-4479-98e8-588fc9a4e2ce/volumes"
Nov 01 22:45:15.077793 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:15.072104    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 01 22:45:15.077793 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:15.072162    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 01 22:45:15.126350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:15.125220    4531 generic.go:356] "Generic (PLEG): container finished" podID="ad9c4429-dac1-4fae-9132-a815c3e3718f" containerID="41f2abbaa359b2f13f8649289b0d629df9ad444ffd4979d9ea4044970d60b4b1" exitCode=0
Nov 01 22:45:15.126350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:15.126292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c" event={"ID":"ad9c4429-dac1-4fae-9132-a815c3e3718f","Type":"ContainerDied","Data":"41f2abbaa359b2f13f8649289b0d629df9ad444ffd4979d9ea4044970d60b4b1"}
Nov 01 22:45:15.809644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:15.809614    4531 scope.go:122] "RemoveContainer" containerID="83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e"
Nov 01 22:45:16.079789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.075157    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 01 22:45:16.079789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.075231    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 01 22:45:16.136476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.136448    4531 generic.go:356] "Generic (PLEG): container finished" podID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerID="b2411db68b24d05bd5d93a28bb144b92d13d1401daf3c36f7f23f2ec7dcd59f4" exitCode=60
Nov 01 22:45:16.136918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.136521    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerDied","Data":"b2411db68b24d05bd5d93a28bb144b92d13d1401daf3c36f7f23f2ec7dcd59f4"}
Nov 01 22:45:16.136918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.136575    4531 scope.go:122] "RemoveContainer" containerID="7e280cf24f88020515cf459fcf7cbec957b261f2cf87099db19086639ad306d0"
Nov 01 22:45:16.137094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.137069    4531 scope.go:122] "RemoveContainer" containerID="b2411db68b24d05bd5d93a28bb144b92d13d1401daf3c36f7f23f2ec7dcd59f4"
Nov 01 22:45:16.137427 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:16.137404    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5_odh-ai-gateway-infra(1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5"
Nov 01 22:45:16.138602 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.138581    4531 generic.go:356] "Generic (PLEG): container finished" podID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerID="e515e519a99e360a247b9dd0fb6758249f9351cf80ad8adcf8d3d54b42b4276b" exitCode=60
Nov 01 22:45:16.138685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.138607    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerDied","Data":"e515e519a99e360a247b9dd0fb6758249f9351cf80ad8adcf8d3d54b42b4276b"}
Nov 01 22:45:16.139066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.139017    4531 scope.go:122] "RemoveContainer" containerID="e515e519a99e360a247b9dd0fb6758249f9351cf80ad8adcf8d3d54b42b4276b"
Nov 01 22:45:16.139257 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:16.139238    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64_odh-ai-gateway-infra(f895a121-9216-45e9-a694-56e7c3d4ac9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a"
Nov 01 22:45:16.151135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.151110    4531 scope.go:122] "RemoveContainer" containerID="57bf5ceb241a6e65f367d1c0e74f0aca403457ff7c641944563d0b8d566b9519"
Nov 01 22:45:16.439223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.439196    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:16.524419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.524387    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5\" (UniqueName: \"kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5\") pod \"ad9c4429-dac1-4fae-9132-a815c3e3718f\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") "
Nov 01 22:45:16.524561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.524451    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca\") pod \"ad9c4429-dac1-4fae-9132-a815c3e3718f\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") "
Nov 01 22:45:16.524658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.524638    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca\") pod \"ad9c4429-dac1-4fae-9132-a815c3e3718f\" (UID: \"ad9c4429-dac1-4fae-9132-a815c3e3718f\") "
Nov 01 22:45:16.525105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.525076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca" pod "ad9c4429-dac1-4fae-9132-a815c3e3718f" (UID: "ad9c4429-dac1-4fae-9132-a815c3e3718f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:16.525380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.525362    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:16.526471 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.526453    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5" pod "ad9c4429-dac1-4fae-9132-a815c3e3718f" (UID: "ad9c4429-dac1-4fae-9132-a815c3e3718f"). InnerVolumeSpecName "kube-api-access-696t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:16.527127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.527102    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca" pod "ad9c4429-dac1-4fae-9132-a815c3e3718f" (UID: "ad9c4429-dac1-4fae-9132-a815c3e3718f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:16.627010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.626965    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-696t5\" (UniqueName: \"kubernetes.io/projected/ad9c4429-dac1-4fae-9132-a815c3e3718f-kube-api-access-696t5\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:16.627010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.627008    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad9c4429-dac1-4fae-9132-a815c3e3718f-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:16.714935 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.714884    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:16.809721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:16.809696    4531 scope.go:122] "RemoveContainer" containerID="c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780"
Nov 01 22:45:17.089750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.083107    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 01 22:45:17.089750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.083167    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 01 22:45:17.176749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.172595    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"
Nov 01 22:45:17.176749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.174920    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c" event={"ID":"ad9c4429-dac1-4fae-9132-a815c3e3718f","Type":"ContainerDied","Data":"91108dd026968136b8b12ca15c74fd238a3d7024f8474ab1803c5db0fe07f9fb"}
Nov 01 22:45:17.176749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.174977    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91108dd026968136b8b12ca15c74fd238a3d7024f8474ab1803c5db0fe07f9fb"
Nov 01 22:45:17.183747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.182808    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerID="ec091b56c28c4287b973a4b352d625574497ad444272494ad04cd45dd7588615" exitCode=60
Nov 01 22:45:17.183747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.182919    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerDied","Data":"ec091b56c28c4287b973a4b352d625574497ad444272494ad04cd45dd7588615"}
Nov 01 22:45:17.183747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.182967    4531 scope.go:122] "RemoveContainer" containerID="83c6f5a1efa9bb84d3eab54582fabf3a3dde408628953060f238a32282ff935e"
Nov 01 22:45:17.183747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.183455    4531 scope.go:122] "RemoveContainer" containerID="ec091b56c28c4287b973a4b352d625574497ad444272494ad04cd45dd7588615"
Nov 01 22:45:17.184066 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:17.183870    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7_odh-ai-gateway-infra(ea053b55-4cca-4af7-a05d-7aff6c5cdf93)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93"
Nov 01 22:45:17.303417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.303335    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"]
Nov 01 22:45:17.405602 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.405563    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"]
Nov 01 22:45:17.618843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.618778    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"]
Nov 01 22:45:17.620080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.620052    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ad9c4429-dac1-4fae-9132-a815c3e3718f" containerName="revoke-keys"
Nov 01 22:45:17.620080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.620081    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad9c4429-dac1-4fae-9132-a815c3e3718f" containerName="revoke-keys"
Nov 01 22:45:17.620315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.620299    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad9c4429-dac1-4fae-9132-a815c3e3718f" containerName="revoke-keys"
Nov 01 22:45:17.628868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.628841    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.631983 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.631141    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:17.694774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.688628    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:17.694774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.692971    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"]
Nov 01 22:45:17.737224 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.737200    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:17.750441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.749707    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk\" (UniqueName: \"kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk\") pod \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") "
Nov 01 22:45:17.750441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.749990    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca\") pod \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\" (UID: \"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5\") "
Nov 01 22:45:17.750441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.750238    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.750441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.750381    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.750721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.750537    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca" pod "1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" (UID: "1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:17.750807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.750781    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9qlh\" (UniqueName: \"kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.750925 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.750906    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:17.761751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.760855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk" pod "1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" (UID: "1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5"). InnerVolumeSpecName "kube-api-access-kc6bk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:17.851647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.851619    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca\") pod \"f895a121-9216-45e9-a694-56e7c3d4ac9a\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") "
Nov 01 22:45:17.851822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.851805    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2\" (UniqueName: \"kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2\") pod \"f895a121-9216-45e9-a694-56e7c3d4ac9a\" (UID: \"f895a121-9216-45e9-a694-56e7c3d4ac9a\") "
Nov 01 22:45:17.851990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.851968    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca" pod "f895a121-9216-45e9-a694-56e7c3d4ac9a" (UID: "f895a121-9216-45e9-a694-56e7c3d4ac9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:17.852201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.852180    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.852430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.852409    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9qlh\" (UniqueName: \"kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.852493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.852457    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.852586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.852573    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc6bk\" (UniqueName: \"kubernetes.io/projected/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5-kube-api-access-kc6bk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:17.852629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.852590    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f895a121-9216-45e9-a694-56e7c3d4ac9a-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:17.853317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.853240    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.855805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.855782    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.924821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.915090    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2" pod "f895a121-9216-45e9-a694-56e7c3d4ac9a" (UID: "f895a121-9216-45e9-a694-56e7c3d4ac9a"). InnerVolumeSpecName "kube-api-access-xdpj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:17.924821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.920934    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9qlh\" (UniqueName: \"kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh\") pod \"maas-api-revoke-sub-d9172dfd-w2kcp\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.943550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.943529    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:17.954445 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:17.954413    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdpj2\" (UniqueName: \"kubernetes.io/projected/f895a121-9216-45e9-a694-56e7c3d4ac9a-kube-api-access-xdpj2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:18.101153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.082321    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 01 22:45:18.101153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.082380    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 01 22:45:18.215892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.212931    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64" event={"ID":"f895a121-9216-45e9-a694-56e7c3d4ac9a","Type":"ContainerDied","Data":"70be786d93fb5695c447a42f39fa95e34ea335601b7c0501493cb2cb79d6681a"}
Nov 01 22:45:18.215892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.213000    4531 scope.go:122] "RemoveContainer" containerID="e515e519a99e360a247b9dd0fb6758249f9351cf80ad8adcf8d3d54b42b4276b"
Nov 01 22:45:18.215892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.213099    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"
Nov 01 22:45:18.240039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.239665    4531 generic.go:356] "Generic (PLEG): container finished" podID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerID="31f720c2ab22a23bdfcbac4d97b581a943a80012a8a4d882faa6b8b6ab34135e" exitCode=60
Nov 01 22:45:18.240039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.239818    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerDied","Data":"31f720c2ab22a23bdfcbac4d97b581a943a80012a8a4d882faa6b8b6ab34135e"}
Nov 01 22:45:18.240632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.240615    4531 scope.go:122] "RemoveContainer" containerID="31f720c2ab22a23bdfcbac4d97b581a943a80012a8a4d882faa6b8b6ab34135e"
Nov 01 22:45:18.241813 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:18.241616    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6_odh-ai-gateway-infra(1b8d5fb4-97c9-4b20-bfc9-849fe95aae33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33"
Nov 01 22:45:18.241869 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.241815    4531 scope.go:122] "RemoveContainer" containerID="c701de7a06a5eb891fa299b83fe50db8920969b7ccd6b4b091bc65172a720780"
Nov 01 22:45:18.244045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.244023    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5" event={"ID":"1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5","Type":"ContainerDied","Data":"e3ba6f0bec0ed00e19df484098fc7a50ebfec28932c465f47ee4d26804235ff7"}
Nov 01 22:45:18.244211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.244199    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"
Nov 01 22:45:18.292715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.292680    4531 scope.go:122] "RemoveContainer" containerID="b2411db68b24d05bd5d93a28bb144b92d13d1401daf3c36f7f23f2ec7dcd59f4"
Nov 01 22:45:18.401561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.401487    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"]
Nov 01 22:45:18.506659 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.506307    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"]
Nov 01 22:45:18.632037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.632006    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:18.635194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.635162    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"]
Nov 01 22:45:18.640778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.640757    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-5b8fd2-29892885-cbl64"]
Nov 01 22:45:18.706412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.706371    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp\" (UniqueName: \"kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp\") pod \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") "
Nov 01 22:45:18.753770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.753743    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"]
Nov 01 22:45:18.766055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.766023    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp" pod "ea053b55-4cca-4af7-a05d-7aff6c5cdf93" (UID: "ea053b55-4cca-4af7-a05d-7aff6c5cdf93"). InnerVolumeSpecName "kube-api-access-nnrqp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:18.780799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.780768    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-d6101b-29892885-bzzm5"]
Nov 01 22:45:18.808011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.807980    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca\") pod \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\" (UID: \"ea053b55-4cca-4af7-a05d-7aff6c5cdf93\") "
Nov 01 22:45:18.808329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.808303    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca" pod "ea053b55-4cca-4af7-a05d-7aff6c5cdf93" (UID: "ea053b55-4cca-4af7-a05d-7aff6c5cdf93"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:18.808926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.808897    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:18.808926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.808927    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnrqp\" (UniqueName: \"kubernetes.io/projected/ea053b55-4cca-4af7-a05d-7aff6c5cdf93-kube-api-access-nnrqp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:18.818424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.818398    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" path="/var/lib/kubelet/pods/1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5/volumes"
Nov 01 22:45:18.818990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:18.818971    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" path="/var/lib/kubelet/pods/f895a121-9216-45e9-a694-56e7c3d4ac9a/volumes"
Nov 01 22:45:19.067958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.067935    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body=
Nov 01 22:45:19.068074 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.067986    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused"
Nov 01 22:45:19.260650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.260615    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp" event={"ID":"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8","Type":"ContainerStarted","Data":"f9220a0d13e40990b4292d9310876cd4731ecc1d6690be465b2db92886eee647"}
Nov 01 22:45:19.261138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.260678    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp" event={"ID":"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8","Type":"ContainerStarted","Data":"abf058d9c32ab71d69499490af591e291c83a469f9bd84ab52d8c6152ff6af25"}
Nov 01 22:45:19.262941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.262907    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"
Nov 01 22:45:19.263071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.262960    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7" event={"ID":"ea053b55-4cca-4af7-a05d-7aff6c5cdf93","Type":"ContainerDied","Data":"b9ec975823d55731d0b20a2c5854c1d38a3863b695053fb9bebb35f744e3c342"}
Nov 01 22:45:19.263071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.263034    4531 scope.go:122] "RemoveContainer" containerID="ec091b56c28c4287b973a4b352d625574497ad444272494ad04cd45dd7588615"
Nov 01 22:45:19.385979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.385852    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp" podStartSLOduration=2.385840071 podStartE2EDuration="2.385840071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:19.380158143 +0000 UTC m=+1300.697808157" watchObservedRunningTime="2026-11-01 22:45:19.385840071 +0000 UTC m=+1300.703490072"
Nov 01 22:45:19.447785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.447753    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"]
Nov 01 22:45:19.508497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.508461    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"]
Nov 01 22:45:19.515551 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.515522    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892885-cvvs7"]
Nov 01 22:45:19.683919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.683887    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:19.726787 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.726757    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9\" (UniqueName: \"kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9\") pod \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") "
Nov 01 22:45:19.727055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.727036    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca\") pod \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\" (UID: \"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33\") "
Nov 01 22:45:19.727533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.727506    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca" pod "1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" (UID: "1b8d5fb4-97c9-4b20-bfc9-849fe95aae33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:19.727656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.727641    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:19.729082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.729060    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9" pod "1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" (UID: "1b8d5fb4-97c9-4b20-bfc9-849fe95aae33"). InnerVolumeSpecName "kube-api-access-dbgp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:19.787314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.787282    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"]
Nov 01 22:45:19.788005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.787989    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.788005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788005    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788017    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788022    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788028    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788034    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788043    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788047    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788061    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788066    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788073    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788078    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788088    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788093    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788117    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788122    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788132    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.788153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788136    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788244    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788252    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788258    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788264    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788273    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788279    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788284    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788290    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788297    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788303    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" containerName="cleanup"
Nov 01 22:45:19.789022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.788309    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:19.793263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.792802    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.831017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.830979    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.831154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.831110    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.831538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.831385    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tbnv\" (UniqueName: \"kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.831614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.831564    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbgp9\" (UniqueName: \"kubernetes.io/projected/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33-kube-api-access-dbgp9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:19.886423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.879591    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"]
Nov 01 22:45:19.932840 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.932763    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tbnv\" (UniqueName: \"kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.933003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.932910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.933003 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.932987    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.934384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.934352    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.939427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.939398    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:19.992618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:19.992587    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tbnv\" (UniqueName: \"kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv\") pod \"maas-api-revoke-sub-6ae58dcd-dt4wp\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:20.141853 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.141825    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:20.283415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.283287    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6" event={"ID":"1b8d5fb4-97c9-4b20-bfc9-849fe95aae33","Type":"ContainerDied","Data":"2934e0b191fdf67461e3bad4fb30e4041de004a73c3d54c55f48b276cc9a80a7"}
Nov 01 22:45:20.283415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.283401    4531 scope.go:122] "RemoveContainer" containerID="31f720c2ab22a23bdfcbac4d97b581a943a80012a8a4d882faa6b8b6ab34135e"
Nov 01 22:45:20.283950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.283548    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"
Nov 01 22:45:20.288805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.288781    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:20.291291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.291105    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:20.293613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.293317    4531 generic.go:356] "Generic (PLEG): container finished" podID="f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" containerID="f9220a0d13e40990b4292d9310876cd4731ecc1d6690be465b2db92886eee647" exitCode=0
Nov 01 22:45:20.293613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.293444    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp" event={"ID":"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8","Type":"ContainerDied","Data":"f9220a0d13e40990b4292d9310876cd4731ecc1d6690be465b2db92886eee647"}
Nov 01 22:45:20.293797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.293613    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:45:20.557718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.557642    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"]
Nov 01 22:45:20.588113 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:20.588079    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee907ae_6811_438b_8ea2_73aca9aca133.slice/crio-719a743051dbf434a3b545deb12690be498bce998effc4c785ea309cd4637e69 WatchSource:0}: Error finding container 719a743051dbf434a3b545deb12690be498bce998effc4c785ea309cd4637e69: Status 404 returned error can't find the container with id 719a743051dbf434a3b545deb12690be498bce998effc4c785ea309cd4637e69
Nov 01 22:45:20.820995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.820933    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea053b55-4cca-4af7-a05d-7aff6c5cdf93" path="/var/lib/kubelet/pods/ea053b55-4cca-4af7-a05d-7aff6c5cdf93/volumes"
Nov 01 22:45:20.887344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.887313    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"]
Nov 01 22:45:20.899111 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:20.899076    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c768f3-29892885-wjct6"]
Nov 01 22:45:21.307783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.307719    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp" event={"ID":"8ee907ae-6811-438b-8ea2-73aca9aca133","Type":"ContainerStarted","Data":"807fc375e5ab821e240834f98a6d091e01c7852b3f308bce7d175e331dd2a638"}
Nov 01 22:45:21.308183 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.307806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp" event={"ID":"8ee907ae-6811-438b-8ea2-73aca9aca133","Type":"ContainerStarted","Data":"719a743051dbf434a3b545deb12690be498bce998effc4c785ea309cd4637e69"}
Nov 01 22:45:21.427777 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.426772    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp" podStartSLOduration=2.426759383 podStartE2EDuration="2.426759383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:21.423999168 +0000 UTC m=+1302.741649170" watchObservedRunningTime="2026-11-01 22:45:21.426759383 +0000 UTC m=+1302.744409391"
Nov 01 22:45:21.505222 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.505197    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:21.559181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.559119    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca\") pod \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") "
Nov 01 22:45:21.559307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.559196    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh\" (UniqueName: \"kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh\") pod \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") "
Nov 01 22:45:21.559419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.559401    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca\") pod \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\" (UID: \"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8\") "
Nov 01 22:45:21.561529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.561505    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh" pod "f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" (UID: "f7b512c2-859d-4ef1-9327-0fa75c5ba9c8"). InnerVolumeSpecName "kube-api-access-b9qlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:21.564474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.564447    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca" pod "f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" (UID: "f7b512c2-859d-4ef1-9327-0fa75c5ba9c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:21.574275 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.574254    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca" pod "f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" (UID: "f7b512c2-859d-4ef1-9327-0fa75c5ba9c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:21.661633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.661603    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:21.661778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.661637    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:21.661778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:21.661652    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9qlh\" (UniqueName: \"kubernetes.io/projected/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8-kube-api-access-b9qlh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:22.266440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.266367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:22.321064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.321033    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"
Nov 01 22:45:22.323159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.321033    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp" event={"ID":"f7b512c2-859d-4ef1-9327-0fa75c5ba9c8","Type":"ContainerDied","Data":"abf058d9c32ab71d69499490af591e291c83a469f9bd84ab52d8c6152ff6af25"}
Nov 01 22:45:22.323697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.323660    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abf058d9c32ab71d69499490af591e291c83a469f9bd84ab52d8c6152ff6af25"
Nov 01 22:45:22.324151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.324125    4531 generic.go:356] "Generic (PLEG): container finished" podID="8ee907ae-6811-438b-8ea2-73aca9aca133" containerID="807fc375e5ab821e240834f98a6d091e01c7852b3f308bce7d175e331dd2a638" exitCode=0
Nov 01 22:45:22.324811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.324784    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp" event={"ID":"8ee907ae-6811-438b-8ea2-73aca9aca133","Type":"ContainerDied","Data":"807fc375e5ab821e240834f98a6d091e01c7852b3f308bce7d175e331dd2a638"}
Nov 01 22:45:22.821482 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:22.821445    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" path="/var/lib/kubelet/pods/1b8d5fb4-97c9-4b20-bfc9-849fe95aae33/volumes"
Nov 01 22:45:23.865760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:23.863615    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:23.972752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:23.962608    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv\" (UniqueName: \"kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv\") pod \"8ee907ae-6811-438b-8ea2-73aca9aca133\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") "
Nov 01 22:45:23.972752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:23.962667    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca\") pod \"8ee907ae-6811-438b-8ea2-73aca9aca133\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") "
Nov 01 22:45:23.972752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:23.962766    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca\") pod \"8ee907ae-6811-438b-8ea2-73aca9aca133\" (UID: \"8ee907ae-6811-438b-8ea2-73aca9aca133\") "
Nov 01 22:45:23.972752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:23.967933    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca" pod "8ee907ae-6811-438b-8ea2-73aca9aca133" (UID: "8ee907ae-6811-438b-8ea2-73aca9aca133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:24.002027 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:23.995007    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv" pod "8ee907ae-6811-438b-8ea2-73aca9aca133" (UID: "8ee907ae-6811-438b-8ea2-73aca9aca133"). InnerVolumeSpecName "kube-api-access-2tbnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:24.023884 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.017335    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca" pod "8ee907ae-6811-438b-8ea2-73aca9aca133" (UID: "8ee907ae-6811-438b-8ea2-73aca9aca133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:24.074799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.066509    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tbnv\" (UniqueName: \"kubernetes.io/projected/8ee907ae-6811-438b-8ea2-73aca9aca133-kube-api-access-2tbnv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:24.074799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.066548    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:24.074799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.066563    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ee907ae-6811-438b-8ea2-73aca9aca133-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:24.207837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.207801    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"]
Nov 01 22:45:24.408796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.408465    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp" event={"ID":"8ee907ae-6811-438b-8ea2-73aca9aca133","Type":"ContainerDied","Data":"719a743051dbf434a3b545deb12690be498bce998effc4c785ea309cd4637e69"}
Nov 01 22:45:24.408796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.408529    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="719a743051dbf434a3b545deb12690be498bce998effc4c785ea309cd4637e69"
Nov 01 22:45:24.408796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.408601    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"
Nov 01 22:45:24.530960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.526903    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8bbdc73c-mrll9"]
Nov 01 22:45:24.883751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.869042    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e9762f0-db50-4768-ab64-5136ceb3d8df" path="/var/lib/kubelet/pods/6e9762f0-db50-4768-ab64-5136ceb3d8df/volumes"
Nov 01 22:45:24.998866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:24.998814    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.148298 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.147008    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.369750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.369561    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.398803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.397804    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.414476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.413301    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:45:25.436409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.435812    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.487847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.487806    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:25.889760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.889080    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:45:25.903751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:25.901130    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.246654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.245101    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.438756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.436884    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:45:26.438756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.437365    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:45:26.439007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.438898    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:45:26.439007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.438953    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:45:26.495836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.493915    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.495836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.494304    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:45:26.889746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.889685    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:26.910750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:26.901873    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.179811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.177487    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.451055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.450996    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.490022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.489981    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.577753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.544468    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.585665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.571870    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.606234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.606190    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.676752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.676104    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:45:27.676752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.676548    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"]
Nov 01 22:45:27.676752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.676671    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:45:27.676752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.676720    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:45:27.714932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.677779    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy" containerID="cri-o://aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e" gracePeriod=30
Nov 01 22:45:27.882972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.882933    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"]
Nov 01 22:45:27.886748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.883406    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" containerID="cri-o://88d54acbd5573517c57ca5af555fdc2b10d1a8f5deece503eea4938d851f6585" gracePeriod=30
Nov 01 22:45:27.891123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.891093    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:27.927753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:27.924421    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.141068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.140983    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:45:28.300715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.300673    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.609289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.609240    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:28.681570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.680032    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.681570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.680874    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.704867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.690799    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.704867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.691200    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:45:28.706122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.706089    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:45:28.709747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.706623    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:45:28.892885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:28.892796    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.151810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.151654    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.331768 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.315842    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.390702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.379575    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:45:29.735752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.725963    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.797767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.783854    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 01 22:45:29.902753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.900002    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.940752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.939838    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:29.948845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:29.948803    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.051995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.044193    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:45:30.112289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.074938    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.112289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.075036    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:45:30.199858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.199246    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:45:30.200362 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.200213    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.212775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.201487    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:45:30.212934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.212825    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:45:30.395477 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.395395    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:45:30.418961 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.418761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:30.523116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.522943    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:45:30.524478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.524454    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:45:30.528773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.528749    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:45:30.536967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.536943    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:45:30.541771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.541749    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"]
Nov 01 22:45:30.547375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.547356    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8ee907ae-6811-438b-8ea2-73aca9aca133" containerName="revoke-keys"
Nov 01 22:45:30.549431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.549413    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ee907ae-6811-438b-8ea2-73aca9aca133" containerName="revoke-keys"
Nov 01 22:45:30.549584 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.549572    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" containerName="revoke-keys"
Nov 01 22:45:30.549686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.549676    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" containerName="revoke-keys"
Nov 01 22:45:30.549837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.549826    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:30.549918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.549908    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f895a121-9216-45e9-a694-56e7c3d4ac9a" containerName="cleanup"
Nov 01 22:45:30.550036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550027    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:30.550115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550106    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea93a65-6d6d-4f38-a32b-4721d7cc2fd5" containerName="cleanup"
Nov 01 22:45:30.550203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550192    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:30.550271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550261    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:30.550749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550719    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ee907ae-6811-438b-8ea2-73aca9aca133" containerName="revoke-keys"
Nov 01 22:45:30.550865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550855    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" containerName="revoke-keys"
Nov 01 22:45:30.550983 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.550973    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b8d5fb4-97c9-4b20-bfc9-849fe95aae33" containerName="cleanup"
Nov 01 22:45:30.561831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.561756    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"]
Nov 01 22:45:30.562060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.561999    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:45:30.568230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.562443    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.615492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.615463    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:45:30.615847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.615801    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:45:30.633352 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.633314    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.633513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.633440    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.633513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.633491    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvkm\" (UniqueName: \"kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.734648 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.734617    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.735566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.734702    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.735566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.734786    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvkm\" (UniqueName: \"kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.740205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.740170    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.741700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.741669    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.835195 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.835150    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvkm\" (UniqueName: \"kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm\") pod \"maas-api-revoke-sub-16899b33-r6998\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:30.947322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:30.944823    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:31.435146 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:31.430541    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"]
Nov 01 22:45:31.539743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:31.539699    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998" event={"ID":"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203","Type":"ContainerStarted","Data":"a64fcb0a3be2e78cb00cc93aadf25f9d513d0073c0231a3abc2ec76623500360"}
Nov 01 22:45:32.554662 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:32.554623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998" event={"ID":"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203","Type":"ContainerStarted","Data":"77497ca59893a920acf24a2b4e985987045075ceb948aa4f8603fde79884ebd9"}
Nov 01 22:45:32.683372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:32.683316    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998" podStartSLOduration=3.683299834 podStartE2EDuration="3.683299834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:32.678031301 +0000 UTC m=+1313.995681303" watchObservedRunningTime="2026-11-01 22:45:32.683299834 +0000 UTC m=+1314.000949836"
Nov 01 22:45:32.978109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:32.978062    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:45:33.058799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.058766    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-nhzgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:45:33.059018 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.058821    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:45:33.085334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085301    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085454 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085408    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085501    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085607    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.085689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085658    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085682    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085816 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085741    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085811    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085849    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.085873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085852    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.086036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.085888    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data\") pod \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\" (UID: \"747f48c3-52de-4ed9-a5b7-ea0d3960fd40\") "
Nov 01 22:45:33.086128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086104    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:33.086377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086354    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.086556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086532    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.086671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086652    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.086819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086691    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.086819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086705    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.086819 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.086718    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.088492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.088470    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "kube-api-access-ff5dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.088492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.088481    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.088651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.088611    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:33.088713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.088683    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token" pod "747f48c3-52de-4ed9-a5b7-ea0d3960fd40" (UID: "747f48c3-52de-4ed9-a5b7-ea0d3960fd40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.188552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.188522    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.188552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.188552    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5dr\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-kube-api-access-ff5dr\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.188552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.188563    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.188808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.188576    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.188808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.188586    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747f48c3-52de-4ed9-a5b7-ea0d3960fd40-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.217766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.217699    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7128-2f383d"]
Nov 01 22:45:33.218481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.218460    4531 cpu_manager.go:414] "containerMap: removing container" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy"
Nov 01 22:45:33.218597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.218482    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy"
Nov 01 22:45:33.218703 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.218688    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy"
Nov 01 22:45:33.218798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.218709    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy"
Nov 01 22:45:33.222850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.222832    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:33.272609 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.272557    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5sq6\""
Nov 01 22:45:33.276436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.276405    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-2f383d"]
Nov 01 22:45:33.289711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.289676    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shkc6\" (UniqueName: \"kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6\") pod \"test-curl-7128-2f383d\" (UID: \"a1e7a3b8-20e4-41c5-b960-b90217195a84\") " pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:33.391542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.391502    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shkc6\" (UniqueName: \"kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6\") pod \"test-curl-7128-2f383d\" (UID: \"a1e7a3b8-20e4-41c5-b960-b90217195a84\") " pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:33.392867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.392844    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7137-e23eae"]
Nov 01 22:45:33.393564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.393548    4531 cpu_manager.go:414] "containerMap: removing container" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy"
Nov 01 22:45:33.393564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.393565    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerName="istio-proxy"
Nov 01 22:45:33.397766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.397711    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:33.456628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.456599    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shkc6\" (UniqueName: \"kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6\") pod \"test-curl-7128-2f383d\" (UID: \"a1e7a3b8-20e4-41c5-b960-b90217195a84\") " pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:33.465017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.462853    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-e23eae"]
Nov 01 22:45:33.493265 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.492961    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fwcv\" (UniqueName: \"kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv\") pod \"test-curl-7137-e23eae\" (UID: \"18d2b7e5-5d70-4cd6-9eea-526876de6421\") " pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:33.538582 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.538546    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:33.592558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.592509    4531 generic.go:356] "Generic (PLEG): container finished" podID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" containerID="aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e" exitCode=0
Nov 01 22:45:33.593008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.592710    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" event={"ID":"747f48c3-52de-4ed9-a5b7-ea0d3960fd40","Type":"ContainerDied","Data":"aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e"}
Nov 01 22:45:33.593008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.592855    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f" event={"ID":"747f48c3-52de-4ed9-a5b7-ea0d3960fd40","Type":"ContainerDied","Data":"8dda84c77d2f0d8770140b8f2b4695768944b434a97e5cacef702ba9af6fc51e"}
Nov 01 22:45:33.593008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.592891    4531 scope.go:122] "RemoveContainer" containerID="aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e"
Nov 01 22:45:33.593250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.593232    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"
Nov 01 22:45:33.599627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.599595    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fwcv\" (UniqueName: \"kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv\") pod \"test-curl-7137-e23eae\" (UID: \"18d2b7e5-5d70-4cd6-9eea-526876de6421\") " pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:33.601293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.601272    4531 generic.go:356] "Generic (PLEG): container finished" podID="b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" containerID="77497ca59893a920acf24a2b4e985987045075ceb948aa4f8603fde79884ebd9" exitCode=0
Nov 01 22:45:33.601585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.601562    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998" event={"ID":"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203","Type":"ContainerDied","Data":"77497ca59893a920acf24a2b4e985987045075ceb948aa4f8603fde79884ebd9"}
Nov 01 22:45:33.603492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.603470    4531 generic.go:356] "Generic (PLEG): container finished" podID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerID="88d54acbd5573517c57ca5af555fdc2b10d1a8f5deece503eea4938d851f6585" exitCode=0
Nov 01 22:45:33.603585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.603545    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" event={"ID":"17c9a213-b1b8-4ae5-8d4a-e15fa591732f","Type":"ContainerDied","Data":"88d54acbd5573517c57ca5af555fdc2b10d1a8f5deece503eea4938d851f6585"}
Nov 01 22:45:33.605442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.605424    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:45:33.620461 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.620441    4531 scope.go:122] "RemoveContainer" containerID="66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"
Nov 01 22:45:33.634811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.634787    4531 scope.go:122] "RemoveContainer" containerID="aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e"
Nov 01 22:45:33.635240 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:33.635190    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e\": container with ID starting with aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e not found: ID does not exist" containerID="aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e"
Nov 01 22:45:33.635389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.635359    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e"} err="failed to get container status \"aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e\": rpc error: code = NotFound desc = could not find container \"aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e\": container with ID starting with aff1b8ffb5d45da2ba65fc071b99d0cba0cade4b9e57e37c6aa444852fe3af7e not found: ID does not exist"
Nov 01 22:45:33.635493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.635481    4531 scope.go:122] "RemoveContainer" containerID="66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"
Nov 01 22:45:33.635935 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:33.635877    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d\": container with ID starting with 66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d not found: ID does not exist" containerID="66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"
Nov 01 22:45:33.635935 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.635908    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d"} err="failed to get container status \"66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d\": rpc error: code = NotFound desc = could not find container \"66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d\": container with ID starting with 66d5970bb58e8b058b2055dbd51a79ff0e7b37294d098ee42899895fae8bb40d not found: ID does not exist"
Nov 01 22:45:33.666992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.666966    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fwcv\" (UniqueName: \"kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv\") pod \"test-curl-7137-e23eae\" (UID: \"18d2b7e5-5d70-4cd6-9eea-526876de6421\") " pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:33.701379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701350    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.701547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701412    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.701547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701464    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.701547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701501    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.701715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701551    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.701715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701623    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.701715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701685    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.702034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701901    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.702034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.701928    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:33.702279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.702235    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.702279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.702243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.703147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.702878    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.703147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.702930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb\") pod \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\" (UID: \"17c9a213-b1b8-4ae5-8d4a-e15fa591732f\") "
Nov 01 22:45:33.703147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.703114    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.703586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.703564    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.703586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.703586    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.703756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.703596    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.703756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.703606    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.703756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.703620    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.704098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.704072    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:45:33.705424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.705383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "kube-api-access-9mkdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.705595 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.705475    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:33.705957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.705927    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo" pod "17c9a213-b1b8-4ae5-8d4a-e15fa591732f" (UID: "17c9a213-b1b8-4ae5-8d4a-e15fa591732f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:45:33.717108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.717089    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:33.759414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.759384    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"]
Nov 01 22:45:33.766799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.766772    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-dss7f"]
Nov 01 22:45:33.808923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.806991    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.808923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.807031    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.808923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.807061    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.808923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.807075    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mkdb\" (UniqueName: \"kubernetes.io/projected/17c9a213-b1b8-4ae5-8d4a-e15fa591732f-kube-api-access-9mkdb\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:33.943950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.943924    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-2f383d"]
Nov 01 22:45:33.948073 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:33.948046    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1e7a3b8_20e4_41c5_b960_b90217195a84.slice/crio-334af5770606da83a1f7af5c713192a3fd017551b2a6addd7557f20d75cdca63 WatchSource:0}: Error finding container 334af5770606da83a1f7af5c713192a3fd017551b2a6addd7557f20d75cdca63: Status 404 returned error can't find the container with id 334af5770606da83a1f7af5c713192a3fd017551b2a6addd7557f20d75cdca63
Nov 01 22:45:33.950163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:33.950143    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:45:34.119081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.119045    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7137-e23eae"]
Nov 01 22:45:34.121861 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:34.121832    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d2b7e5_5d70_4cd6_9eea_526876de6421.slice/crio-e6182dd8c212874ee3f3abf6dad133562366cb2bad6de1339a981b26e6ebcf62 WatchSource:0}: Error finding container e6182dd8c212874ee3f3abf6dad133562366cb2bad6de1339a981b26e6ebcf62: Status 404 returned error can't find the container with id e6182dd8c212874ee3f3abf6dad133562366cb2bad6de1339a981b26e6ebcf62
Nov 01 22:45:34.619072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.619037    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-2f383d" event={"ID":"a1e7a3b8-20e4-41c5-b960-b90217195a84","Type":"ContainerStarted","Data":"334af5770606da83a1f7af5c713192a3fd017551b2a6addd7557f20d75cdca63"}
Nov 01 22:45:34.620776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.620720    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"
Nov 01 22:45:34.620776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.620759    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt" event={"ID":"17c9a213-b1b8-4ae5-8d4a-e15fa591732f","Type":"ContainerDied","Data":"d5d83966844f5890283e78ea6001199d80f7590f092c0a56a90acc1bca1a789a"}
Nov 01 22:45:34.620972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.620822    4531 scope.go:122] "RemoveContainer" containerID="88d54acbd5573517c57ca5af555fdc2b10d1a8f5deece503eea4938d851f6585"
Nov 01 22:45:34.622164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.622137    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-e23eae" event={"ID":"18d2b7e5-5d70-4cd6-9eea-526876de6421","Type":"ContainerStarted","Data":"e6182dd8c212874ee3f3abf6dad133562366cb2bad6de1339a981b26e6ebcf62"}
Nov 01 22:45:34.790520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.790460    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"]
Nov 01 22:45:34.796212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.796095    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-nhzgt"]
Nov 01 22:45:34.819556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.819533    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" path="/var/lib/kubelet/pods/17c9a213-b1b8-4ae5-8d4a-e15fa591732f/volumes"
Nov 01 22:45:34.820034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.820019    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747f48c3-52de-4ed9-a5b7-ea0d3960fd40" path="/var/lib/kubelet/pods/747f48c3-52de-4ed9-a5b7-ea0d3960fd40/volumes"
Nov 01 22:45:34.835707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.835686    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:34.923648 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.923576    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca\") pod \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") "
Nov 01 22:45:34.923792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.923704    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca\") pod \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") "
Nov 01 22:45:34.923954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.923934    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca" pod "b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" (UID: "b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:45:34.923999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.923945    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm\" (UniqueName: \"kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm\") pod \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\" (UID: \"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203\") "
Nov 01 22:45:34.924894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.924867    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:34.926213 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.926185    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca" pod "b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" (UID: "b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:45:34.926213 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.926195    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm" pod "b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" (UID: "b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203"). InnerVolumeSpecName "kube-api-access-tpvkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:34.934685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:34.934664    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:35.026457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:35.026420    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvkm\" (UniqueName: \"kubernetes.io/projected/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-kube-api-access-tpvkm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:35.026457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:35.026460    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:35.537487 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:35.537453    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:35.642372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:35.642333    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998" event={"ID":"b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203","Type":"ContainerDied","Data":"a64fcb0a3be2e78cb00cc93aadf25f9d513d0073c0231a3abc2ec76623500360"}
Nov 01 22:45:35.642372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:35.642356    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"
Nov 01 22:45:35.643116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:35.642406    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a64fcb0a3be2e78cb00cc93aadf25f9d513d0073c0231a3abc2ec76623500360"
Nov 01 22:45:38.683230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:38.683148    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7137-e23eae" event={"ID":"18d2b7e5-5d70-4cd6-9eea-526876de6421","Type":"ContainerStarted","Data":"3a042dfe72c3624146436d0b8bd94f9c0610ef709212b00857e3dff54515497f"}
Nov 01 22:45:38.685044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:38.685017    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-2f383d" event={"ID":"a1e7a3b8-20e4-41c5-b960-b90217195a84","Type":"ContainerStarted","Data":"702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f"}
Nov 01 22:45:38.791111 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:38.791057    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7137-e23eae" podStartSLOduration=1.593415245 podStartE2EDuration="5.791041369s" totalImagesPullingTime="4.197626124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:38.789347471 +0000 UTC m=+1320.106997486" watchObservedRunningTime="2026-11-01 22:45:38.791041369 +0000 UTC m=+1320.108691371"
Nov 01 22:45:38.899674 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:38.899618    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7128-2f383d" podStartSLOduration=1.5242662729999998 podStartE2EDuration="5.899600898s" totalImagesPullingTime="4.375334625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:38.89579036 +0000 UTC m=+1320.213440362" watchObservedRunningTime="2026-11-01 22:45:38.899600898 +0000 UTC m=+1320.217250900"
Nov 01 22:45:39.202154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:39.202124    4531 scope.go:122] "RemoveContainer" containerID="09bb93e83ff85a27db7ee909716d3ddff4dd30680f9c932a4ed428c0dfaf4782"
Nov 01 22:45:39.212932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:39.212910    4531 scope.go:122] "RemoveContainer" containerID="fa385500b44e94b026dd2e9b62ab30f6d2671418d5c3f833e2356796d6e30a44"
Nov 01 22:45:40.054343 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.054311    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"]
Nov 01 22:45:40.098378 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.098343    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c3b972d-629zv"]
Nov 01 22:45:40.219652 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.219621    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7137-e23eae"]
Nov 01 22:45:40.235862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.235826    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7137-e23eae"]
Nov 01 22:45:40.525691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.525636    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:40.563878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.563843    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7128-2f383d"]
Nov 01 22:45:40.571515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.571277    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7128-2f383d"]
Nov 01 22:45:40.702971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.702930    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7128-2f383d" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" containerName="test-curl-7128-2f383d" containerID="cri-o://702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f" gracePeriod=2
Nov 01 22:45:40.703137 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.702973    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7137-e23eae" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" containerName="test-curl-7137-e23eae" containerID="cri-o://3a042dfe72c3624146436d0b8bd94f9c0610ef709212b00857e3dff54515497f" gracePeriod=2
Nov 01 22:45:40.750810 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:40.750775    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-2f383d\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:40.798313 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:40.798252    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-e23eae\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:40.818353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:40.818321    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70dfce34-c0ef-4860-b1cf-a78b9b184374" path="/var/lib/kubelet/pods/70dfce34-c0ef-4860-b1cf-a78b9b184374/volumes"
Nov 01 22:45:40.846159 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:40.846128    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-2f383d\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:40.894562 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:40.894527    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-e23eae\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:41.366230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:41.366202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:42.101240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:42.101190    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:42.514442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:42.514380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:42.543526 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:42.543490    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:45:42.728018 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:42.727988    4531 generic.go:356] "Generic (PLEG): container finished" podID="18d2b7e5-5d70-4cd6-9eea-526876de6421" containerID="3a042dfe72c3624146436d0b8bd94f9c0610ef709212b00857e3dff54515497f" exitCode=137
Nov 01 22:45:42.894237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:42.893610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:43.032933 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.032868    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:43.036684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.036658    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:43.091173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.091144    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv\" (UniqueName: \"kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv\") pod \"18d2b7e5-5d70-4cd6-9eea-526876de6421\" (UID: \"18d2b7e5-5d70-4cd6-9eea-526876de6421\") "
Nov 01 22:45:43.091307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.091287    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6\" (UniqueName: \"kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6\") pod \"a1e7a3b8-20e4-41c5-b960-b90217195a84\" (UID: \"a1e7a3b8-20e4-41c5-b960-b90217195a84\") "
Nov 01 22:45:43.093425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.093385    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv" pod "18d2b7e5-5d70-4cd6-9eea-526876de6421" (UID: "18d2b7e5-5d70-4cd6-9eea-526876de6421"). InnerVolumeSpecName "kube-api-access-8fwcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:43.096772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.094001    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6" pod "a1e7a3b8-20e4-41c5-b960-b90217195a84" (UID: "a1e7a3b8-20e4-41c5-b960-b90217195a84"). InnerVolumeSpecName "kube-api-access-shkc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:45:43.192960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.192929    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fwcv\" (UniqueName: \"kubernetes.io/projected/18d2b7e5-5d70-4cd6-9eea-526876de6421-kube-api-access-8fwcv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:43.192960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.192957    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shkc6\" (UniqueName: \"kubernetes.io/projected/a1e7a3b8-20e4-41c5-b960-b90217195a84-kube-api-access-shkc6\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:45:43.199779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.199750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:43.684959 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.684920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:43.742111 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.742076    4531 generic.go:356] "Generic (PLEG): container finished" podID="a1e7a3b8-20e4-41c5-b960-b90217195a84" containerID="702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f" exitCode=137
Nov 01 22:45:43.742299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.742123    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:43.742299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.742174    4531 scope.go:122] "RemoveContainer" containerID="702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f"
Nov 01 22:45:43.744079 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.744054    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:43.754100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.754084    4531 scope.go:122] "RemoveContainer" containerID="702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f"
Nov 01 22:45:43.754369 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:43.754351    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f\": container with ID starting with 702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f not found: ID does not exist" containerID="702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f"
Nov 01 22:45:43.754417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.754378    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f"} err="failed to get container status \"702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f\": rpc error: code = NotFound desc = could not find container \"702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f\": container with ID starting with 702c88eaa150c2b16c5676ff624aa435ec98c19bf47f66f300e35c3e6c735b4f not found: ID does not exist"
Nov 01 22:45:43.754417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:43.754395    4531 scope.go:122] "RemoveContainer" containerID="3a042dfe72c3624146436d0b8bd94f9c0610ef709212b00857e3dff54515497f"
Nov 01 22:45:43.799666 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:43.799627    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-2f383d\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:43.892327 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:43.892290    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-2f383d\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" pod="opendatahub/test-curl-7128-2f383d"
Nov 01 22:45:43.939982 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:45:43.939902    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7137-e23eae\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" pod="opendatahub/test-curl-7137-e23eae"
Nov 01 22:45:44.015479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:44.015449    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:44.833548 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:44.833512    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" path="/var/lib/kubelet/pods/18d2b7e5-5d70-4cd6-9eea-526876de6421/volumes"
Nov 01 22:45:44.834808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:44.834766    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" path="/var/lib/kubelet/pods/a1e7a3b8-20e4-41c5-b960-b90217195a84/volumes"
Nov 01 22:45:45.189522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.189439    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"]
Nov 01 22:45:45.190244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190224    4531 cpu_manager.go:414] "containerMap: removing container" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy"
Nov 01 22:45:45.190244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190244    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy"
Nov 01 22:45:45.190425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190273    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" containerName="test-curl-7128-2f383d"
Nov 01 22:45:45.190425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190280    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" containerName="test-curl-7128-2f383d"
Nov 01 22:45:45.190425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190324    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" containerName="revoke-keys"
Nov 01 22:45:45.190425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190331    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" containerName="revoke-keys"
Nov 01 22:45:45.190425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190366    4531 cpu_manager.go:414] "containerMap: removing container" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" containerName="test-curl-7137-e23eae"
Nov 01 22:45:45.190425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190373    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" containerName="test-curl-7137-e23eae"
Nov 01 22:45:45.190714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190550    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c9a213-b1b8-4ae5-8d4a-e15fa591732f" containerName="istio-proxy"
Nov 01 22:45:45.190714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190566    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1e7a3b8-20e4-41c5-b960-b90217195a84" containerName="test-curl-7128-2f383d"
Nov 01 22:45:45.190714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190584    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" containerName="revoke-keys"
Nov 01 22:45:45.190714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.190597    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d2b7e5-5d70-4cd6-9eea-526876de6421" containerName="test-curl-7137-e23eae"
Nov 01 22:45:45.195812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.195791    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.237630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.237598    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.237789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.237657    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.237911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.237884    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.238070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.237953    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq4rm\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.238070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.238004    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.238070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.238033    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.238070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.238067    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.238288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.238143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.238288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.238193    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.281841 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.280507    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"]
Nov 01 22:45:45.339797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.339756    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.339968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.339843    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.339968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.339906    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.339968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.339946    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340117    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340176    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq4rm\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340225    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340254    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340274    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340579 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340559    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.340664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.340642    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.341316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.341288    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.341514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.341496    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.343404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.343377    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.343825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.343806    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.398015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.397990    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq4rm\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.398248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.398226    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token\") pod \"e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") " pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.510597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.510565    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:45.689223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:45.689184    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:46.007039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.007003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:46.022796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.022771    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"]
Nov 01 22:45:46.024649 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:46.024623    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b0a533_ebfb_4a51_8898_60d0c00e4748.slice/crio-a2f5d0f40a8c56004009ea5cfab96ef87f8ffe925c31e64768ed1f399a801f64 WatchSource:0}: Error finding container a2f5d0f40a8c56004009ea5cfab96ef87f8ffe925c31e64768ed1f399a801f64: Status 404 returned error can't find the container with id a2f5d0f40a8c56004009ea5cfab96ef87f8ffe925c31e64768ed1f399a801f64
Nov 01 22:45:46.027310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.027279    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:46.027434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.027362    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:46.027434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.027390    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:46.345789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.345650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:46.616322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.616243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:46.803203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.803169    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" event={"ID":"85b0a533-ebfb-4a51-8898-60d0c00e4748","Type":"ContainerStarted","Data":"0b5b9035f4d435910e03f08f87f43944443e3505a8338144b5018fe4afae0f71"}
Nov 01 22:45:46.803379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.803239    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" event={"ID":"85b0a533-ebfb-4a51-8898-60d0c00e4748","Type":"ContainerStarted","Data":"a2f5d0f40a8c56004009ea5cfab96ef87f8ffe925c31e64768ed1f399a801f64"}
Nov 01 22:45:46.927450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:46.927350    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podStartSLOduration=1.927336333 podStartE2EDuration="1.927336333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:46.925135705 +0000 UTC m=+1328.242785716" watchObservedRunningTime="2026-11-01 22:45:46.927336333 +0000 UTC m=+1328.244986334"
Nov 01 22:45:47.510947 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:47.510907    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:47.513168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:47.513139    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 01 22:45:47.513303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:47.513194    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 01 22:45:47.986245 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:47.986203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:48.342636 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:48.342561    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:48.511188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:48.511154    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 01 22:45:48.511597 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:48.511221    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 01 22:45:49.511031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:49.510989    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 01 22:45:49.511250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:49.511228    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 01 22:45:50.511545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:50.511508    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 01 22:45:50.512081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:50.511566    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 01 22:45:51.181184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.181147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:51.426882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.426846    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:51.511635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.511580    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 01 22:45:51.512181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.511661    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 01 22:45:51.722028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.719499    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:51.946482 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.946431    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"]
Nov 01 22:45:51.956631 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.956565    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:51.959201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:51.959170    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"]
Nov 01 22:45:52.018824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.018792    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d5014777\""
Nov 01 22:45:52.019302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.019279    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d5014777-dockercfg-ndhz5\""
Nov 01 22:45:52.081321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.081284    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9f4c\" (UniqueName: \"kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c\") pod \"payload-processing-e2e-ait-d5014777-dd6976c65-68hhw\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") " pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.081532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.081512    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume\") pod \"payload-processing-e2e-ait-d5014777-dd6976c65-68hhw\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") " pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.089455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.089427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:52.183208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.183176    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9f4c\" (UniqueName: \"kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c\") pod \"payload-processing-e2e-ait-d5014777-dd6976c65-68hhw\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") " pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.183415 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.183360    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume\") pod \"payload-processing-e2e-ait-d5014777-dd6976c65-68hhw\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") " pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.184361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.184337    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume\") pod \"payload-processing-e2e-ait-d5014777-dd6976c65-68hhw\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") " pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.239055 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.238971    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9f4c\" (UniqueName: \"kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c\") pod \"payload-processing-e2e-ait-d5014777-dd6976c65-68hhw\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") " pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.300952 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.300919    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:52.374363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.372132    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:52.511239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.511151    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body=
Nov 01 22:45:52.511239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.511212    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused"
Nov 01 22:45:52.693365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.693256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:52.781172 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:52.777200    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"]
Nov 01 22:45:52.781172 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:52.777511    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86b15181_8769_4a4f_bcfe_b2f6d1110747.slice/crio-b9e067ec206955bff8aaef7c21810ca6c42c89ef3e94b0262e3f0cdd3fab14ec WatchSource:0}: Error finding container b9e067ec206955bff8aaef7c21810ca6c42c89ef3e94b0262e3f0cdd3fab14ec: Status 404 returned error can't find the container with id b9e067ec206955bff8aaef7c21810ca6c42c89ef3e94b0262e3f0cdd3fab14ec
Nov 01 22:45:53.051517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:53.050214    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw" event={"ID":"86b15181-8769-4a4f-bcfe-b2f6d1110747","Type":"ContainerStarted","Data":"d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b"}
Nov 01 22:45:53.051517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:53.050283    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw" event={"ID":"86b15181-8769-4a4f-bcfe-b2f6d1110747","Type":"ContainerStarted","Data":"b9e067ec206955bff8aaef7c21810ca6c42c89ef3e94b0262e3f0cdd3fab14ec"}
Nov 01 22:45:53.051517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:53.051404    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:45:53.164749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:53.164459    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:53.314803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:53.294955    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw" podStartSLOduration=2.294936688 podStartE2EDuration="2.294936688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:53.251086541 +0000 UTC m=+1334.568736566" watchObservedRunningTime="2026-11-01 22:45:53.294936688 +0000 UTC m=+1334.612586688"
Nov 01 22:45:53.715477 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:53.715292    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:54.010021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:54.009093    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:54.010021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:54.009810    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:54.010977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:54.010951    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:45:54.028635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:54.028606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:57.035667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.035629    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"]
Nov 01 22:45:57.040611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.040588    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.136738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.136820    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.136849    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.136922    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5xk\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.136987    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.137031    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.137067    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.137127    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.138784 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.137159    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.171409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.171363    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"]
Nov 01 22:45:57.183383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.183352    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:57.247991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.247926    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248028    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5xk\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248103    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248138    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248218 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248170    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248231    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248264    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248467    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248523    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.248972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.248941    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.249206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.249185    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.249506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.249465    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.249950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.249925    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.250312 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.250290    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.256559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.256057    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.257391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.257366    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.317139 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.317059    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.323915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.323888    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5xk\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk\") pod \"e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") " pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.358018 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.357982    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:57.435979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.435942    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:57.448521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.448492    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"]
Nov 01 22:45:57.454651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.454625    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.524835 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.514698    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-798f4550\""
Nov 01 22:45:57.553904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.553859    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"]
Nov 01 22:45:57.559240 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.559201    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.559491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.559470    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4qfp\" (UniqueName: \"kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp\") pod \"payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.626501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.626423    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"]
Nov 01 22:45:57.632271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.632244    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.661831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.661796    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.661989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.661876    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4qfp\" (UniqueName: \"kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp\") pod \"payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.662897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.662859    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.683591 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.683565    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-798f4550-dockercfg-jt4dx\""
Nov 01 22:45:57.705885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.705854    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"]
Nov 01 22:45:57.718155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.718132    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4qfp\" (UniqueName: \"kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp\") pod \"payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.755560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.755528    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:57.763197 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.763166    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume\") pod \"payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") " pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.763397 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.763380    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsw97\" (UniqueName: \"kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97\") pod \"payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") " pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.785394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.784418    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:57.866459 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.864839    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume\") pod \"payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") " pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.866459 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.865164    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsw97\" (UniqueName: \"kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97\") pod \"payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") " pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.866459 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.866397    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume\") pod \"payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") " pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.902324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.902230    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"]
Nov 01 22:45:57.947893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.943485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsw97\" (UniqueName: \"kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97\") pod \"payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") " pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.960905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.960880    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:57.981928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.981868    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:57.982058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.981972    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:57.982058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:57.982015    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:45:58.045068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.042913    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:45:58.150070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.150037    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" event={"ID":"86dec67a-933b-46da-b3dd-5cfc53f02093","Type":"ContainerStarted","Data":"2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d"}
Nov 01 22:45:58.150219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.150102    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" event={"ID":"86dec67a-933b-46da-b3dd-5cfc53f02093","Type":"ContainerStarted","Data":"5a484660d9cd9a9e022d11375b0ed7d9eb66b4c4bdc6b284c235bbf6f615560b"}
Nov 01 22:45:58.269806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.269721    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podStartSLOduration=2.26970694 podStartE2EDuration="2.26970694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:58.26711698 +0000 UTC m=+1339.584766983" watchObservedRunningTime="2026-11-01 22:45:58.26970694 +0000 UTC m=+1339.587356940"
Nov 01 22:45:58.333214 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:58.333172    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c67740c_1c80_47bd_8f03_495d6269dfbf.slice/crio-d9907a1bca011cd637da95cdd34d86d00fd50ca588260e239da8786de3c1da7d WatchSource:0}: Error finding container d9907a1bca011cd637da95cdd34d86d00fd50ca588260e239da8786de3c1da7d: Status 404 returned error can't find the container with id d9907a1bca011cd637da95cdd34d86d00fd50ca588260e239da8786de3c1da7d
Nov 01 22:45:58.333858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.333755    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"]
Nov 01 22:45:58.360050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.360023    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:45:58.427267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:58.427239    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"]
Nov 01 22:45:58.432840 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:45:58.432818    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod129d0479_1fbc_4f27_a779_eab25243f675.slice/crio-1b86d6c3c7375fd87357feeeccaff9228734361c0e37328cec5ef7ff1564faa5 WatchSource:0}: Error finding container 1b86d6c3c7375fd87357feeeccaff9228734361c0e37328cec5ef7ff1564faa5: Status 404 returned error can't find the container with id 1b86d6c3c7375fd87357feeeccaff9228734361c0e37328cec5ef7ff1564faa5
Nov 01 22:45:59.179751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.179208    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k" event={"ID":"129d0479-1fbc-4f27-a779-eab25243f675","Type":"ContainerStarted","Data":"d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c"}
Nov 01 22:45:59.179751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.179276    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k" event={"ID":"129d0479-1fbc-4f27-a779-eab25243f675","Type":"ContainerStarted","Data":"1b86d6c3c7375fd87357feeeccaff9228734361c0e37328cec5ef7ff1564faa5"}
Nov 01 22:45:59.179751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.179334    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:45:59.193478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.192631    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2" event={"ID":"6c67740c-1c80-47bd-8f03-495d6269dfbf","Type":"ContainerStarted","Data":"9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622"}
Nov 01 22:45:59.193478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.192708    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2" event={"ID":"6c67740c-1c80-47bd-8f03-495d6269dfbf","Type":"ContainerStarted","Data":"d9907a1bca011cd637da95cdd34d86d00fd50ca588260e239da8786de3c1da7d"}
Nov 01 22:45:59.193478 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.192765    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:45:59.310751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.310666    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k" podStartSLOduration=2.310650535 podStartE2EDuration="2.310650535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:59.306925684 +0000 UTC m=+1340.624575685" watchObservedRunningTime="2026-11-01 22:45:59.310650535 +0000 UTC m=+1340.628300558"
Nov 01 22:45:59.359699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.359670    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:45:59.359897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.359768    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:45:59.439432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:45:59.439335    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2" podStartSLOduration=2.4393223920000002 podStartE2EDuration="2.439322392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:45:59.433983789 +0000 UTC m=+1340.751633790" watchObservedRunningTime="2026-11-01 22:45:59.439322392 +0000 UTC m=+1340.756972392"
Nov 01 22:46:00.358986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:00.358954    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:00.359364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:00.359012    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:01.358375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:01.358336    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:01.358540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:01.358393    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:02.359651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:02.359086    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:02.359651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:02.359147    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:02.768751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:02.765353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:03.088823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.087860    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:03.362807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.360186    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:03.362807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.360264    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:03.816751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.811474    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:03.850750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.832861    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:03.879748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.872576    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.922030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.921986    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:03.937316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:03.937271    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.130812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.122194    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"]
Nov 01 22:46:04.189750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.188158    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:04.222751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.213572    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.278377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.278334    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"]
Nov 01 22:46:04.299766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.280540    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b50b3403\""
Nov 01 22:46:04.305949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.305919    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b50b3403-dockercfg-lbqrc\""
Nov 01 22:46:04.308147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.308112    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.308496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.308475    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprl2\" (UniqueName: \"kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2\") pod \"payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.399154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.399057    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.469449 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.399912    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:04.469701 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.469673    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:04.486862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.436172    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.487102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.435251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume\") pod \"payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.487401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.487377    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprl2\" (UniqueName: \"kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2\") pod \"payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.603752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.579350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:04.626751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.588880    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.772752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.752860    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.772752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.753180    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.821753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.801519    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprl2\" (UniqueName: \"kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2\") pod \"payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") " pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:04.850774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.850720    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:04.872898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.872851    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:04.978899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.978831    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.342264 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.988105    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.372754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:04.988549    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:05.402882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.021474    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.402882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.402355    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:05.402882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.402423    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:05.402882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.192680    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.440774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.130085    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.525751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.494064    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.652799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.648344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:05.757764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.750290    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:46:05.757764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.750373    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:46:05.757764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.750428    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:46:05.757764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.750480    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:46:05.757764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.750532    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:46:05.758303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.749565    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.833751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.811843    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:46:05.833751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.812303    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:46:05.876661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.853798    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:05.876661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.854580    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.876661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.866025    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:46:05.876661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.866086    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:46:05.876661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.873694    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:05.893761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:05.878244    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.049886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.015606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:06.360806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.346683    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.373756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.373625    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:06.373756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.373683    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:06.385962 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.385833    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:06.385962 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.385893    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.457752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.433814    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.457752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.434405    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 01 22:46:06.457752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.434622    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:46:06.510739 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.510592    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:06.647815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.646486    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:46:06.814459 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.814416    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.069036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.882961    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.069036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.884817    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.069036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:06.887894    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.202605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.202558    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": context deadline exceeded"
Nov 01 22:46:07.432816 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.389771    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.455829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.390854    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:07.456129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.456077    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:07.456290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.441845    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.456738 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.441808    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.534945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.534896    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.799808 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.730487    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.828878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.828830    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.905992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.905946    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:07.907088 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.907056    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:07.907975 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:07.907951    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.359893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.359857    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:08.360162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.360135    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:08.372453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.372343    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"]
Nov 01 22:46:08.457357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.457319    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.458687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.457587    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.479860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.479816    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 01 22:46:08.535843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.535802    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.589926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.589899    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:46:08.735538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.730948    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.735538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.731489    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:46:08.839387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.839024    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.907792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.906821    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.914649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.914613    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:08.983315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.983277    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:08.998847 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:08.998803    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.497323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.497274    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.498039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.498005    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.498531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.498499    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.530804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.530413    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:09.530804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.530473    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:09.549991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.549926    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.577929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.577895    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:09.973753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.950876    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.973753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.746767    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.973753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.899425    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": context deadline exceeded"
Nov 01 22:46:09.973753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.932342    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:09.973753 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:46:09.972208    4531 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.159s"
Nov 01 22:46:09.973753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.972284    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v" event={"ID":"abee2b83-ee15-4da9-ac76-8ed6a4272b8d","Type":"ContainerStarted","Data":"95ec30927d926e01e22182e50104939ee3b49804dbb72c89d333672474bfd76f"}
Nov 01 22:46:09.980761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:09.932580    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.046893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.046832    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.129764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.118584    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.138856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.138823    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:46:10.139094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.139066    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.367752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.352489    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:46:10.406807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.380060    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:10.406807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.380116    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:10.434430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.434398    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:46:10.525329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.525282    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/health\": context deadline exceeded"
Nov 01 22:46:10.546036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.545954    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.546388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.546317    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.546618 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.546592    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.806271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.806233    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/health\": context deadline exceeded"
Nov 01 22:46:10.892016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.891970    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.978814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.959785    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:10.978814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.962789    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": context deadline exceeded"
Nov 01 22:46:10.979496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:10.979465    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.014274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.014211    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.071059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.070957    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.387996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.387261    4531 patch_prober.go:29] interesting pod/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused" start-of-body=
Nov 01 22:46:11.387996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.387319    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.141:15021/healthz/ready\": dial tcp 10.133.0.141:15021: connect: connection refused"
Nov 01 22:46:11.547517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.547476    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.550535 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.548125    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:11.590533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.590502    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:46:11.641084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.641015    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 22:46:11.641308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.641279    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:11.641502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.641487    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 22:46:11.641641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.641629    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 22:46:11.641801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.641787    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 22:46:11.641941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.641929    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 22:46:11.642067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.642057    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 22:46:11.655075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.655041    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 22:46:11.677004 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.676949    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v" podStartSLOduration=8.676932104 podStartE2EDuration="8.676932104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:11.655911636 +0000 UTC m=+1352.973561638" watchObservedRunningTime="2026-11-01 22:46:11.676932104 +0000 UTC m=+1352.994582108"
Nov 01 22:46:11.678177 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.678154    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 22:46:11.690417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:11.690397    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 22:46:12.358291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:12.358262    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:46:12.410188 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:12.410157    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v" event={"ID":"abee2b83-ee15-4da9-ac76-8ed6a4272b8d","Type":"ContainerStarted","Data":"a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602"}
Nov 01 22:46:12.426119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:12.425884    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:46:12.427396 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:12.427370    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 22:46:12.543126 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:46:12.543085    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:46:13.542594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:13.542549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:13.832790 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:13.832670    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:14.224487 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:14.224455    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:14.782646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:14.782610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:18.938807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:18.938694    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:20.412404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:20.412374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:20.720523 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:20.720491    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:20.971579 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:20.971513    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:22.139909 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.139876    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"]
Nov 01 22:46:22.145958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.145937    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.261396 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.261365    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"]
Nov 01 22:46:22.264413 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.264390    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 22:46:22.264563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.264542    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 22:46:22.264642 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.264584    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 22:46:22.264709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.264671    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-b55rk\""
Nov 01 22:46:22.300785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.300752    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.300965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.300840    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.300965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.300885    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.301104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.301018    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4xg\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.301104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.301053    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.301104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.301072    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.301264 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.301226    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402458    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402511    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402566    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402593    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402703    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4xg\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402789    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.402833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.402827    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.403405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.403377    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.405201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.405180    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.405302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.405213    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.405302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.405253    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.405302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.405268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.460446 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.460418    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.460446 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.460431    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4xg\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gzf6j\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:22.578118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.578088    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:22.759494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:22.759466    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:23.245920 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:46:23.245889    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04f8d576_38b7_4f1e_aaa7_6b6434ade817.slice/crio-86be1d2a35dbf83d8ec0667ab1a7ce5a36a21b6919824084871d3aab6290ddf4 WatchSource:0}: Error finding container 86be1d2a35dbf83d8ec0667ab1a7ce5a36a21b6919824084871d3aab6290ddf4: Status 404 returned error can't find the container with id 86be1d2a35dbf83d8ec0667ab1a7ce5a36a21b6919824084871d3aab6290ddf4
Nov 01 22:46:23.246950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:23.246926    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"]
Nov 01 22:46:23.427635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:23.427610    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:46:23.527025 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:23.526935    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" event={"ID":"04f8d576-38b7-4f1e-aaa7-6b6434ade817","Type":"ContainerStarted","Data":"86be1d2a35dbf83d8ec0667ab1a7ce5a36a21b6919824084871d3aab6290ddf4"}
Nov 01 22:46:23.864679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:23.864592    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:26.816123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:26.816026    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:27.342446 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.342404    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"]
Nov 01 22:46:27.348401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.348375    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.386900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.386758    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.386900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.386805    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.386900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.386877    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.387176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.386957    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.387176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.387125    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.387176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.387167    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d62p\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.387318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.387218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.387318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.387259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.387389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.387323    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.467176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.467142    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"]
Nov 01 22:46:27.489198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489158    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d62p\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489244    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489285    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489361    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489438    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489553    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489581    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.489863 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.489788    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.490243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.490218    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.490496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.490475    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.491150 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.491124    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.491432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.491404    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.491546 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.491519    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.495532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.495509    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.496227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.496177    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.544691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.544669    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d62p\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.546383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.546360    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-sl6fs\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.668125 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.667872    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:27.980516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.980474    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:27.980946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:27.980562    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:28.151017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.150973    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"]
Nov 01 22:46:28.156669 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:46:28.156635    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b7dd35_ea1b_4285_88cf_fc5b404039f3.slice/crio-fb4e3fb4df07ca650823a37c8d0744f08f0401643fc3da2507a658e946c174b8 WatchSource:0}: Error finding container fb4e3fb4df07ca650823a37c8d0744f08f0401643fc3da2507a658e946c174b8: Status 404 returned error can't find the container with id fb4e3fb4df07ca650823a37c8d0744f08f0401643fc3da2507a658e946c174b8
Nov 01 22:46:28.159482 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.159444    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:28.159583 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.159525    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:28.159583 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.159574    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:28.587119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.586150    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" event={"ID":"04f8d576-38b7-4f1e-aaa7-6b6434ade817","Type":"ContainerStarted","Data":"d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5"}
Nov 01 22:46:28.587119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.586598    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:28.588145 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.588118    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" event={"ID":"80b7dd35-ea1b-4285-88cf-fc5b404039f3","Type":"ContainerStarted","Data":"7b0d6dd705674572a99557d4d6679b1610110ba2a328da4926520f1500257b4b"}
Nov 01 22:46:28.588272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.588170    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" event={"ID":"80b7dd35-ea1b-4285-88cf-fc5b404039f3","Type":"ContainerStarted","Data":"fb4e3fb4df07ca650823a37c8d0744f08f0401643fc3da2507a658e946c174b8"}
Nov 01 22:46:28.668677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.668646    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:28.698794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.698717    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" podStartSLOduration=1.96775017 podStartE2EDuration="6.698700269s" totalImagesPullingTime="4.730950099s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:28.698008 +0000 UTC m=+1370.015658002" watchObservedRunningTime="2026-11-01 22:46:28.698700269 +0000 UTC m=+1370.016350271"
Nov 01 22:46:28.814779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.814703    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podStartSLOduration=1.814685666 podStartE2EDuration="1.814685666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:28.812908311 +0000 UTC m=+1370.130558337" watchObservedRunningTime="2026-11-01 22:46:28.814685666 +0000 UTC m=+1370.132335668"
Nov 01 22:46:28.941104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:28.941019    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:29.439758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:29.439478    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:29.607364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:29.607234    4531 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gzf6j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:46:29.607364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:29.607277    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" podUID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:46:29.680752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:29.678833    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:29.680752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:29.678887    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:30.679752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:30.673428    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:30.679752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:30.673483    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:31.672754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:31.671083    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:31.672754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:31.671139    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:32.606337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:32.606310    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:46:32.680057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:32.680021    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:32.680308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:32.680281    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:33.679110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:33.679074    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:33.680379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:33.680348    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:34.669143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:34.669110    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:34.669318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:34.669171    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:35.668426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:35.668393    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-sl6fs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused" start-of-body=
Nov 01 22:46:35.668957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:35.668444    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.146:15021/healthz/ready\": dial tcp 10.133.0.146:15021: connect: connection refused"
Nov 01 22:46:36.611181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:36.611140    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:36.850773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:36.850692    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:37.181909 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:37.181879    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:37.182512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:37.182486    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:37.184292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:37.184272    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 22:46:39.200469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.200432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:39.293288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.293253    4531 scope.go:122] "RemoveContainer" containerID="d07c19dec23cd6624eb6a449259d29e591ff24a4e8eeb2c5c2f947f8f4134af1"
Nov 01 22:46:39.308002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.307966    4531 scope.go:122] "RemoveContainer" containerID="555074d756240851cde31f8d7f64738064dd458b61b9b95c8e73f6a1019f3e6d"
Nov 01 22:46:39.320164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.320144    4531 scope.go:122] "RemoveContainer" containerID="4c990c246ce477b523f899dffdd13e0ebb75bd670daa2808806c40800749b547"
Nov 01 22:46:39.333189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.333165    4531 scope.go:122] "RemoveContainer" containerID="095e5358311020ba56dea06ac4389b7a8ec01e48e553456423279d7c639cf90d"
Nov 01 22:46:39.344203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.344180    4531 scope.go:122] "RemoveContainer" containerID="feb6bcbfaa9a503671f232f11f8476ca4fb8eef94bfc6e084bbe194f01c76f00"
Nov 01 22:46:39.353572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.353556    4531 scope.go:122] "RemoveContainer" containerID="cbb6a9a78ffb36f010515ec91c34539732b19c9aa8277c87e705521bbeb17891"
Nov 01 22:46:39.363305 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:39.363271    4531 scope.go:122] "RemoveContainer" containerID="847fd533c15c33f8044894fe7d9d9ac6e92279b6e1e2db41588fe3f36a3df248"
Nov 01 22:46:42.475766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.467835    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"]
Nov 01 22:46:42.487919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.481790    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:46:42.552910    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knhdb\" (UniqueName: \"kubernetes.io/projected/d344e962-a243-4156-b9aa-353322a53d47-kube-api-access-knhdb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560471    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d344e962-a243-4156-b9aa-353322a53d47-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560535    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560560    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560597    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560619    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d344e962-a243-4156-b9aa-353322a53d47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560665    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560780    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d344e962-a243-4156-b9aa-353322a53d47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.561782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.560905    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.606820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.605506    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"]
Nov 01 22:46:42.667044 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667005    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knhdb\" (UniqueName: \"kubernetes.io/projected/d344e962-a243-4156-b9aa-353322a53d47-kube-api-access-knhdb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667061    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d344e962-a243-4156-b9aa-353322a53d47-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667159    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667198    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d344e962-a243-4156-b9aa-353322a53d47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667268    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667545 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667398    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d344e962-a243-4156-b9aa-353322a53d47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.667660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.667543    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.669058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.668089    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.669058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.668493    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.669058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.668975    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d344e962-a243-4156-b9aa-353322a53d47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.669308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.669154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.669308 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.669211    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.671858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.671812    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d344e962-a243-4156-b9aa-353322a53d47-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.672882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.672837    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d344e962-a243-4156-b9aa-353322a53d47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.754359 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.743575    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knhdb\" (UniqueName: \"kubernetes.io/projected/d344e962-a243-4156-b9aa-353322a53d47-kube-api-access-knhdb\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.754359 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.745872    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d344e962-a243-4156-b9aa-353322a53d47-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-mdd8f\" (UID: \"d344e962-a243-4156-b9aa-353322a53d47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:42.839755 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:42.825268    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:43.667891 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:46:43.667855    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd344e962_a243_4156_b9aa_353322a53d47.slice/crio-b15df71fb16f6a4d4c7920125dd82765bf322f14c4989870d377db2d8e9c4f58 WatchSource:0}: Error finding container b15df71fb16f6a4d4c7920125dd82765bf322f14c4989870d377db2d8e9c4f58: Status 404 returned error can't find the container with id b15df71fb16f6a4d4c7920125dd82765bf322f14c4989870d377db2d8e9c4f58
Nov 01 22:46:43.701820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:43.701780    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:43.702052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:43.702026    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:43.702210 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:43.702189    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:46:43.702886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:43.702845    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"]
Nov 01 22:46:43.843110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:43.843075    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" event={"ID":"d344e962-a243-4156-b9aa-353322a53d47","Type":"ContainerStarted","Data":"b15df71fb16f6a4d4c7920125dd82765bf322f14c4989870d377db2d8e9c4f58"}
Nov 01 22:46:44.554882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.554834    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.79:8000/ready\": context deadline exceeded"
Nov 01 22:46:44.625751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.615761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:44.827757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.823299    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.854761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.836256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:44.919476 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.902147    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:44.938502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.938465    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" event={"ID":"d344e962-a243-4156-b9aa-353322a53d47","Type":"ContainerStarted","Data":"044a8c1b9ccd4a31789d5eba9d2ee2cda49ea9e2c6229044009743f8c95ac022"}
Nov 01 22:46:44.978865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:44.978826    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" probeResult="failure" output="Get \"https://10.133.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.050877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:45.050802    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.103506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:45.091682    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podStartSLOduration=3.091663331 podStartE2EDuration="3.091663331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:46:45.06545171 +0000 UTC m=+1386.383101707" watchObservedRunningTime="2026-11-01 22:46:45.091663331 +0000 UTC m=+1386.409313333"
Nov 01 22:46:45.560539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:45.559974    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" probeResult="failure" output="Get \"https://10.133.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:45.844803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:45.842141    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:45.844803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:45.842207    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:45.844803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:45.842477    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:46.874756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:46.872849    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:46.874756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:46.872911    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:46.905781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:46.899281    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:46:47.173320 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:47.173228    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/health\": context deadline exceeded"
Nov 01 22:46:47.827994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:47.827958    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:47.828171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:47.828019    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:48.828557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:48.828527    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:48.829063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:48.829026    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:49.827429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:49.827341    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:49.827429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:49.827402    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:50.055179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:50.055148    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:50.827999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:50.827966    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:50.828198 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:50.828034    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:51.827910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:51.827878    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:51.828391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:51.827944    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:52.836752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:52.835248    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-mdd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused" start-of-body=
Nov 01 22:46:52.836752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:52.835334    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f" podUID="d344e962-a243-4156-b9aa-353322a53d47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.147:15021/healthz/ready\": dial tcp 10.133.0.147:15021: connect: connection refused"
Nov 01 22:46:54.766392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:54.766352    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:54.767361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:54.767337    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:54.768022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:54.768004    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-mdd8f"
Nov 01 22:46:54.943441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:54.943410    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:55.213058 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:55.213021    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:55.457497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:55.457463    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:46:59.948081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:46:59.948041    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:05.066531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:05.066477    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:10.066864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:10.066829    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:12.528113 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:47:12.528080    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:47:15.061465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:15.061425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:20.017360 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:20.017320    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:24.868779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:24.868712    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:29.888429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:29.888388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:35.041053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:35.040991    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:35.681875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:35.681842    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:39.188643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.188602    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"]
Nov 01 22:47:39.194521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.194492    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.207303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.207276    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.207438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.207325    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgb7t\" (UniqueName: \"kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.207526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.207504    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.248440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.248407    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"]
Nov 01 22:47:39.286115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.286085    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:47:39.286296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.286140    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:47:39.310457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.310429    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.310646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.310631    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.310712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.310661    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgb7t\" (UniqueName: \"kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.311627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.311606    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.314270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.314249    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.365386 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.365360    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgb7t\" (UniqueName: \"kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t\") pod \"maas-api-revoke-keys-e2e-ait-d5014777-lqswc\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.511063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.511035    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:39.910760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:39.910719    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"]
Nov 01 22:47:39.912775 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:47:39.912749    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7094636_6118_4728_b2b8_0fecc6d3e07d.slice/crio-bdf59ad8de75d5813e170b735501ab13d5e3fb3b9ce51a96de28786b5845ff85 WatchSource:0}: Error finding container bdf59ad8de75d5813e170b735501ab13d5e3fb3b9ce51a96de28786b5845ff85: Status 404 returned error can't find the container with id bdf59ad8de75d5813e170b735501ab13d5e3fb3b9ce51a96de28786b5845ff85
Nov 01 22:47:40.228920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:40.228887    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:40.726096 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:40.726057    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc" event={"ID":"b7094636-6118-4728-b2b8-0fecc6d3e07d","Type":"ContainerStarted","Data":"066ef917444310e02fa3c3b64f2f928bb5955b9c176a18fbe61620e99fae6683"}
Nov 01 22:47:40.726288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:40.726113    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc" event={"ID":"b7094636-6118-4728-b2b8-0fecc6d3e07d","Type":"ContainerStarted","Data":"bdf59ad8de75d5813e170b735501ab13d5e3fb3b9ce51a96de28786b5845ff85"}
Nov 01 22:47:40.838858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:40.838801    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc" podStartSLOduration=1.838785984 podStartE2EDuration="1.838785984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:47:40.834992822 +0000 UTC m=+1442.152642805" watchObservedRunningTime="2026-11-01 22:47:40.838785984 +0000 UTC m=+1442.156435979"
Nov 01 22:47:41.739836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:41.739801    4531 generic.go:356] "Generic (PLEG): container finished" podID="b7094636-6118-4728-b2b8-0fecc6d3e07d" containerID="066ef917444310e02fa3c3b64f2f928bb5955b9c176a18fbe61620e99fae6683" exitCode=0
Nov 01 22:47:41.740232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:41.739866    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc" event={"ID":"b7094636-6118-4728-b2b8-0fecc6d3e07d","Type":"ContainerDied","Data":"066ef917444310e02fa3c3b64f2f928bb5955b9c176a18fbe61620e99fae6683"}
Nov 01 22:47:42.531059 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:47:42.531017    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:47:42.946017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:42.945994    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:43.062116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.062085    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca\") pod \"b7094636-6118-4728-b2b8-0fecc6d3e07d\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") "
Nov 01 22:47:43.062274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.062236    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca\") pod \"b7094636-6118-4728-b2b8-0fecc6d3e07d\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") "
Nov 01 22:47:43.062333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.062317    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t\" (UniqueName: \"kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t\") pod \"b7094636-6118-4728-b2b8-0fecc6d3e07d\" (UID: \"b7094636-6118-4728-b2b8-0fecc6d3e07d\") "
Nov 01 22:47:43.062480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.062451    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca" pod "b7094636-6118-4728-b2b8-0fecc6d3e07d" (UID: "b7094636-6118-4728-b2b8-0fecc6d3e07d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:47:43.062898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.062876    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:43.064966 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.064936    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca" pod "b7094636-6118-4728-b2b8-0fecc6d3e07d" (UID: "b7094636-6118-4728-b2b8-0fecc6d3e07d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:47:43.065090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.064966    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t" pod "b7094636-6118-4728-b2b8-0fecc6d3e07d" (UID: "b7094636-6118-4728-b2b8-0fecc6d3e07d"). InnerVolumeSpecName "kube-api-access-zgb7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:47:43.164707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.164671    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7094636-6118-4728-b2b8-0fecc6d3e07d-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:43.164707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.164702    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgb7t\" (UniqueName: \"kubernetes.io/projected/b7094636-6118-4728-b2b8-0fecc6d3e07d-kube-api-access-zgb7t\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:43.757992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.757963    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"
Nov 01 22:47:43.757992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.757984    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc" event={"ID":"b7094636-6118-4728-b2b8-0fecc6d3e07d","Type":"ContainerDied","Data":"bdf59ad8de75d5813e170b735501ab13d5e3fb3b9ce51a96de28786b5845ff85"}
Nov 01 22:47:43.758205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:43.758048    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdf59ad8de75d5813e170b735501ab13d5e3fb3b9ce51a96de28786b5845ff85"
Nov 01 22:47:45.425924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:45.425892    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:47.061264 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.061234    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"]
Nov 01 22:47:47.107632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.107606    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05d3d22d-tx9fv"]
Nov 01 22:47:47.744483 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.744447    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"]
Nov 01 22:47:47.745303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.745280    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b7094636-6118-4728-b2b8-0fecc6d3e07d" containerName="revoke-keys"
Nov 01 22:47:47.745303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.745303    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7094636-6118-4728-b2b8-0fecc6d3e07d" containerName="revoke-keys"
Nov 01 22:47:47.745572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.745504    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7094636-6118-4728-b2b8-0fecc6d3e07d" containerName="revoke-keys"
Nov 01 22:47:47.749643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.749623    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.835306 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.835270    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.835494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.835474    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.835594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.835579    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrms\" (UniqueName: \"kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.845864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.845838    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:47:47.845864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.845855    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:47:47.852144 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.852119    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"]
Nov 01 22:47:47.936887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.936850    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.937046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.936968    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrms\" (UniqueName: \"kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.937174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.937156    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.937751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.937714    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:47.939162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:47.939137    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:48.000641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:48.000585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrms\" (UniqueName: \"kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms\") pod \"maas-api-revoke-keys-e2e-ait-798f4550-d2zmw\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:48.063638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:48.063618    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:48.450355 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:48.450330    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"]
Nov 01 22:47:48.451741 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:47:48.451703    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c89c60f_2648_4745_b189_4353a915b1f1.slice/crio-45ff2bc3cf93130e41f042060620db7da331f6facb23823f1caa7a3d4af081f7 WatchSource:0}: Error finding container 45ff2bc3cf93130e41f042060620db7da331f6facb23823f1caa7a3d4af081f7: Status 404 returned error can't find the container with id 45ff2bc3cf93130e41f042060620db7da331f6facb23823f1caa7a3d4af081f7
Nov 01 22:47:48.818252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:48.818226    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251c836c-f80a-497a-ae5e-a381c12bf8d9" path="/var/lib/kubelet/pods/251c836c-f80a-497a-ae5e-a381c12bf8d9/volumes"
Nov 01 22:47:48.818587 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:48.818568    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw" event={"ID":"3c89c60f-2648-4745-b189-4353a915b1f1","Type":"ContainerStarted","Data":"45ff2bc3cf93130e41f042060620db7da331f6facb23823f1caa7a3d4af081f7"}
Nov 01 22:47:49.822463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:49.822427    4531 generic.go:356] "Generic (PLEG): container finished" podID="3c89c60f-2648-4745-b189-4353a915b1f1" containerID="405f6e1e2a90ee63f59cfe42630a74a668f9106eac8549643c98e3b9f9137f94" exitCode=0
Nov 01 22:47:49.822861 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:49.822478    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw" event={"ID":"3c89c60f-2648-4745-b189-4353a915b1f1","Type":"ContainerDied","Data":"405f6e1e2a90ee63f59cfe42630a74a668f9106eac8549643c98e3b9f9137f94"}
Nov 01 22:47:50.542219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:50.542167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:50.765900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:50.765871    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:50.818717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:50.818652    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"]
Nov 01 22:47:50.973082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:50.973051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:51.026213 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.026184    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:51.187553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.187469    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca\") pod \"3c89c60f-2648-4745-b189-4353a915b1f1\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") "
Nov 01 22:47:51.187553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.187534    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca\") pod \"3c89c60f-2648-4745-b189-4353a915b1f1\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") "
Nov 01 22:47:51.187801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.187576    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms\" (UniqueName: \"kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms\") pod \"3c89c60f-2648-4745-b189-4353a915b1f1\" (UID: \"3c89c60f-2648-4745-b189-4353a915b1f1\") "
Nov 01 22:47:51.187917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.187891    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca" pod "3c89c60f-2648-4745-b189-4353a915b1f1" (UID: "3c89c60f-2648-4745-b189-4353a915b1f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:47:51.188380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.188355    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:51.190541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.190519    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca" pod "3c89c60f-2648-4745-b189-4353a915b1f1" (UID: "3c89c60f-2648-4745-b189-4353a915b1f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:47:51.191605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.191583    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms" pod "3c89c60f-2648-4745-b189-4353a915b1f1" (UID: "3c89c60f-2648-4745-b189-4353a915b1f1"). InnerVolumeSpecName "kube-api-access-rsrms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:47:51.290017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.289988    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c89c60f-2648-4745-b189-4353a915b1f1-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:51.290017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.290016    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrms\" (UniqueName: \"kubernetes.io/projected/3c89c60f-2648-4745-b189-4353a915b1f1-kube-api-access-rsrms\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:51.320743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.320696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:51.538002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.537889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:51.840206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.840142    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"
Nov 01 22:47:51.840206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.840188    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw" event={"ID":"3c89c60f-2648-4745-b189-4353a915b1f1","Type":"ContainerDied","Data":"45ff2bc3cf93130e41f042060620db7da331f6facb23823f1caa7a3d4af081f7"}
Nov 01 22:47:51.840384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:51.840237    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45ff2bc3cf93130e41f042060620db7da331f6facb23823f1caa7a3d4af081f7"
Nov 01 22:47:53.200225 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.200189    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"]
Nov 01 22:47:53.200956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.200940    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3c89c60f-2648-4745-b189-4353a915b1f1" containerName="revoke-keys"
Nov 01 22:47:53.201019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.200956    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c89c60f-2648-4745-b189-4353a915b1f1" containerName="revoke-keys"
Nov 01 22:47:53.201169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.201157    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c89c60f-2648-4745-b189-4353a915b1f1" containerName="revoke-keys"
Nov 01 22:47:53.205229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.205212    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.303537 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.303506    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:47:53.303775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.303578    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:47:53.312814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.312782    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"]
Nov 01 22:47:53.341228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.341201    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.341364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.341271    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.341448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.341368    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9nkf\" (UniqueName: \"kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.442571 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.442539    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.442743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.442612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.442743 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.442650    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9nkf\" (UniqueName: \"kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.443219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.443193    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.444930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.444909    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.496774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.496709    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9nkf\" (UniqueName: \"kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf\") pod \"maas-api-revoke-sub-f5a3e0d9-xkps7\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.519600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.519579    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:53.916014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:53.915984    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"]
Nov 01 22:47:53.918069 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:47:53.918044    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db21044_330e_4979_b3d3_61bce6111a3c.slice/crio-3ddc43d0cd7bd6008d91d4bb0d9063c26dbf77f3c738983079ae572bcfeae233 WatchSource:0}: Error finding container 3ddc43d0cd7bd6008d91d4bb0d9063c26dbf77f3c738983079ae572bcfeae233: Status 404 returned error can't find the container with id 3ddc43d0cd7bd6008d91d4bb0d9063c26dbf77f3c738983079ae572bcfeae233
Nov 01 22:47:54.878622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:54.878587    4531 generic.go:356] "Generic (PLEG): container finished" podID="4db21044-330e-4979-b3d3-61bce6111a3c" containerID="e1af48c69fe5c2583856afd2774386897fdb015c981a063947d8e3bf1aa40d93" exitCode=0
Nov 01 22:47:54.879006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:54.878719    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7" event={"ID":"4db21044-330e-4979-b3d3-61bce6111a3c","Type":"ContainerDied","Data":"e1af48c69fe5c2583856afd2774386897fdb015c981a063947d8e3bf1aa40d93"}
Nov 01 22:47:54.879006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:54.878794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7" event={"ID":"4db21044-330e-4979-b3d3-61bce6111a3c","Type":"ContainerStarted","Data":"3ddc43d0cd7bd6008d91d4bb0d9063c26dbf77f3c738983079ae572bcfeae233"}
Nov 01 22:47:55.819537 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:55.819492    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw" podUID="86b15181-8769-4a4f-bcfe-b2f6d1110747" containerName="payload-processing" containerID="cri-o://d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b" gracePeriod=25
Nov 01 22:47:56.094028 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.093964    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:56.175338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.175311    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:47:56.200307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200277    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume\") pod \"86b15181-8769-4a4f-bcfe-b2f6d1110747\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") "
Nov 01 22:47:56.200444 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200392    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c\" (UniqueName: \"kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c\") pod \"86b15181-8769-4a4f-bcfe-b2f6d1110747\" (UID: \"86b15181-8769-4a4f-bcfe-b2f6d1110747\") "
Nov 01 22:47:56.200515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200468    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca\") pod \"4db21044-330e-4979-b3d3-61bce6111a3c\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") "
Nov 01 22:47:56.200568 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200543    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca\") pod \"4db21044-330e-4979-b3d3-61bce6111a3c\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") "
Nov 01 22:47:56.200656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200628    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume" pod "86b15181-8769-4a4f-bcfe-b2f6d1110747" (UID: "86b15181-8769-4a4f-bcfe-b2f6d1110747"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:47:56.200761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200639    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf\" (UniqueName: \"kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf\") pod \"4db21044-330e-4979-b3d3-61bce6111a3c\" (UID: \"4db21044-330e-4979-b3d3-61bce6111a3c\") "
Nov 01 22:47:56.200894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.200871    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca" pod "4db21044-330e-4979-b3d3-61bce6111a3c" (UID: "4db21044-330e-4979-b3d3-61bce6111a3c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:47:56.202307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.201767    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86b15181-8769-4a4f-bcfe-b2f6d1110747-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:56.202307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.201796    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:56.203126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.203104    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca" pod "4db21044-330e-4979-b3d3-61bce6111a3c" (UID: "4db21044-330e-4979-b3d3-61bce6111a3c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:47:56.203126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.203113    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf" pod "4db21044-330e-4979-b3d3-61bce6111a3c" (UID: "4db21044-330e-4979-b3d3-61bce6111a3c"). InnerVolumeSpecName "kube-api-access-h9nkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:47:56.203255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.203140    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c" pod "86b15181-8769-4a4f-bcfe-b2f6d1110747" (UID: "86b15181-8769-4a4f-bcfe-b2f6d1110747"). InnerVolumeSpecName "kube-api-access-n9f4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:47:56.305279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.305226    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:47:56.306669 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.306642    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9f4c\" (UniqueName: \"kubernetes.io/projected/86b15181-8769-4a4f-bcfe-b2f6d1110747-kube-api-access-n9f4c\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:56.306941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.306670    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4db21044-330e-4979-b3d3-61bce6111a3c-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:56.306941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.306686    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9nkf\" (UniqueName: \"kubernetes.io/projected/4db21044-330e-4979-b3d3-61bce6111a3c-kube-api-access-h9nkf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:47:56.903961 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.903932    4531 generic.go:356] "Generic (PLEG): container finished" podID="86b15181-8769-4a4f-bcfe-b2f6d1110747" containerID="d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b" exitCode=0
Nov 01 22:47:56.904112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.903984    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"
Nov 01 22:47:56.904112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.904012    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw" event={"ID":"86b15181-8769-4a4f-bcfe-b2f6d1110747","Type":"ContainerDied","Data":"d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b"}
Nov 01 22:47:56.904112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.904070    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw" event={"ID":"86b15181-8769-4a4f-bcfe-b2f6d1110747","Type":"ContainerDied","Data":"b9e067ec206955bff8aaef7c21810ca6c42c89ef3e94b0262e3f0cdd3fab14ec"}
Nov 01 22:47:56.904112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.904098    4531 scope.go:122] "RemoveContainer" containerID="d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b"
Nov 01 22:47:56.905953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.905931    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7" event={"ID":"4db21044-330e-4979-b3d3-61bce6111a3c","Type":"ContainerDied","Data":"3ddc43d0cd7bd6008d91d4bb0d9063c26dbf77f3c738983079ae572bcfeae233"}
Nov 01 22:47:56.905953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.905953    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"
Nov 01 22:47:56.906098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.905976    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ddc43d0cd7bd6008d91d4bb0d9063c26dbf77f3c738983079ae572bcfeae233"
Nov 01 22:47:56.918066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.918049    4531 scope.go:122] "RemoveContainer" containerID="d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b"
Nov 01 22:47:56.918347 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:47:56.918330    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b\": container with ID starting with d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b not found: ID does not exist" containerID="d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b"
Nov 01 22:47:56.918404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:56.918355    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b"} err="failed to get container status \"d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b\": rpc error: code = NotFound desc = could not find container \"d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b\": container with ID starting with d2f2e7aa460588b3fd5ff9338d87909e106b8a31a300473c4776e776b402399b not found: ID does not exist"
Nov 01 22:47:57.066008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:57.065979    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"]
Nov 01 22:47:57.112334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:57.112267    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d5014777-dd6976c65-68hhw"]
Nov 01 22:47:58.822021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:47:58.821987    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86b15181-8769-4a4f-bcfe-b2f6d1110747" path="/var/lib/kubelet/pods/86b15181-8769-4a4f-bcfe-b2f6d1110747/volumes"
Nov 01 22:48:00.260692 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.260658    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"]
Nov 01 22:48:00.261550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.261529    4531 cpu_manager.go:414] "containerMap: removing container" podUID="86b15181-8769-4a4f-bcfe-b2f6d1110747" containerName="payload-processing"
Nov 01 22:48:00.261550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.261550    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86b15181-8769-4a4f-bcfe-b2f6d1110747" containerName="payload-processing"
Nov 01 22:48:00.261702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.261566    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4db21044-330e-4979-b3d3-61bce6111a3c" containerName="revoke-keys"
Nov 01 22:48:00.261702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.261571    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db21044-330e-4979-b3d3-61bce6111a3c" containerName="revoke-keys"
Nov 01 22:48:00.261702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.261672    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="86b15181-8769-4a4f-bcfe-b2f6d1110747" containerName="payload-processing"
Nov 01 22:48:00.261702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.261681    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db21044-330e-4979-b3d3-61bce6111a3c" containerName="revoke-keys"
Nov 01 22:48:00.265987 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.265968    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.320336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.320306    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"]
Nov 01 22:48:00.359596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.359551    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:48:00.359837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.359629    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:48:00.359944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.359826    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.360046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.359975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.360046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.360010    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rp8j\" (UniqueName: \"kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.461271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.461240    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.461433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.461288    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.461433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.461322    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rp8j\" (UniqueName: \"kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.462221 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.462197    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.464297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.464271    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.515973 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.515910    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rp8j\" (UniqueName: \"kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j\") pod \"maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.581563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.581533    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:00.587824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.587788    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"]
Nov 01 22:48:00.691871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.691842    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"]
Nov 01 22:48:00.967698 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:00.967672    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"]
Nov 01 22:48:00.969434 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:48:00.969403    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6586ef74_40db_4827_8f20_94915dfe2140.slice/crio-ac7ad550cf90819c0dc7ee8c8cf8222fa02d7cb4b29d05947229b35ca0a7210e WatchSource:0}: Error finding container ac7ad550cf90819c0dc7ee8c8cf8222fa02d7cb4b29d05947229b35ca0a7210e: Status 404 returned error can't find the container with id ac7ad550cf90819c0dc7ee8c8cf8222fa02d7cb4b29d05947229b35ca0a7210e
Nov 01 22:48:01.023766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:01.022158    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:01.221674 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:01.221590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:01.410322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:01.410294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:01.849207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:01.849121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:01.959753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:01.959696    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz" event={"ID":"6586ef74-40db-4827-8f20-94915dfe2140","Type":"ContainerStarted","Data":"bbc35ebf975128d63b17c7879ad0f5741e25d5e4753fe134eec752dc9948a9c9"}
Nov 01 22:48:01.959970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:01.959805    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz" event={"ID":"6586ef74-40db-4827-8f20-94915dfe2140","Type":"ContainerStarted","Data":"ac7ad550cf90819c0dc7ee8c8cf8222fa02d7cb4b29d05947229b35ca0a7210e"}
Nov 01 22:48:02.051421 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:02.051394    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"]
Nov 01 22:48:02.061002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:02.060974    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41f39c92-zckkp"]
Nov 01 22:48:02.108960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:02.108803    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz" podStartSLOduration=2.108784365 podStartE2EDuration="2.108784365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:48:02.073884729 +0000 UTC m=+1463.391534730" watchObservedRunningTime="2026-11-01 22:48:02.108784365 +0000 UTC m=+1463.426434366"
Nov 01 22:48:02.831697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:02.831657    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eec3f78e-c1f2-42ec-81e6-21ba7829f0a7" path="/var/lib/kubelet/pods/eec3f78e-c1f2-42ec-81e6-21ba7829f0a7/volumes"
Nov 01 22:48:02.974489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:02.974455    4531 generic.go:356] "Generic (PLEG): container finished" podID="6586ef74-40db-4827-8f20-94915dfe2140" containerID="bbc35ebf975128d63b17c7879ad0f5741e25d5e4753fe134eec752dc9948a9c9" exitCode=0
Nov 01 22:48:02.974696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:02.974544    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz" event={"ID":"6586ef74-40db-4827-8f20-94915dfe2140","Type":"ContainerDied","Data":"bbc35ebf975128d63b17c7879ad0f5741e25d5e4753fe134eec752dc9948a9c9"}
Nov 01 22:48:04.060862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.060630    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:04.164157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.164127    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:04.223321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.223292    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca\") pod \"6586ef74-40db-4827-8f20-94915dfe2140\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") "
Nov 01 22:48:04.223514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.223486    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j\" (UniqueName: \"kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j\") pod \"6586ef74-40db-4827-8f20-94915dfe2140\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") "
Nov 01 22:48:04.223580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.223556    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca\") pod \"6586ef74-40db-4827-8f20-94915dfe2140\" (UID: \"6586ef74-40db-4827-8f20-94915dfe2140\") "
Nov 01 22:48:04.223944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.223921    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca" pod "6586ef74-40db-4827-8f20-94915dfe2140" (UID: "6586ef74-40db-4827-8f20-94915dfe2140"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:04.224229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.224205    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:04.225557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.225530    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j" pod "6586ef74-40db-4827-8f20-94915dfe2140" (UID: "6586ef74-40db-4827-8f20-94915dfe2140"). InnerVolumeSpecName "kube-api-access-2rp8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:04.225663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.225587    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca" pod "6586ef74-40db-4827-8f20-94915dfe2140" (UID: "6586ef74-40db-4827-8f20-94915dfe2140"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:48:04.325944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.325855    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rp8j\" (UniqueName: \"kubernetes.io/projected/6586ef74-40db-4827-8f20-94915dfe2140-kube-api-access-2rp8j\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:04.325944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.325884    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6586ef74-40db-4827-8f20-94915dfe2140-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:04.996455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.996422    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"
Nov 01 22:48:04.996611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.996449    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz" event={"ID":"6586ef74-40db-4827-8f20-94915dfe2140","Type":"ContainerDied","Data":"ac7ad550cf90819c0dc7ee8c8cf8222fa02d7cb4b29d05947229b35ca0a7210e"}
Nov 01 22:48:04.996611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:04.996511    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac7ad550cf90819c0dc7ee8c8cf8222fa02d7cb4b29d05947229b35ca0a7210e"
Nov 01 22:48:05.588331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.588259    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k" podUID="129d0479-1fbc-4f27-a779-eab25243f675" containerName="payload-processing" containerID="cri-o://d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c" gracePeriod=25
Nov 01 22:48:05.692383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.692342    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2" podUID="6c67740c-1c80-47bd-8f03-495d6269dfbf" containerName="payload-pre-processing" containerID="cri-o://9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622" gracePeriod=25
Nov 01 22:48:05.909510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.909484    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:48:05.948561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.948521    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97\" (UniqueName: \"kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97\") pod \"129d0479-1fbc-4f27-a779-eab25243f675\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") "
Nov 01 22:48:05.948718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.948615    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume\") pod \"129d0479-1fbc-4f27-a779-eab25243f675\" (UID: \"129d0479-1fbc-4f27-a779-eab25243f675\") "
Nov 01 22:48:05.949632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.949585    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume" pod "129d0479-1fbc-4f27-a779-eab25243f675" (UID: "129d0479-1fbc-4f27-a779-eab25243f675"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:05.949774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.949697    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/129d0479-1fbc-4f27-a779-eab25243f675-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:05.975365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:05.973074    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97" pod "129d0479-1fbc-4f27-a779-eab25243f675" (UID: "129d0479-1fbc-4f27-a779-eab25243f675"). InnerVolumeSpecName "kube-api-access-rsw97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:06.009643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.009621    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:48:06.010917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.010889    4531 generic.go:356] "Generic (PLEG): container finished" podID="129d0479-1fbc-4f27-a779-eab25243f675" containerID="d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c" exitCode=0
Nov 01 22:48:06.011042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.010938    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"
Nov 01 22:48:06.011042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.010964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k" event={"ID":"129d0479-1fbc-4f27-a779-eab25243f675","Type":"ContainerDied","Data":"d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c"}
Nov 01 22:48:06.011042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.011029    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k" event={"ID":"129d0479-1fbc-4f27-a779-eab25243f675","Type":"ContainerDied","Data":"1b86d6c3c7375fd87357feeeccaff9228734361c0e37328cec5ef7ff1564faa5"}
Nov 01 22:48:06.011193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.011050    4531 scope.go:122] "RemoveContainer" containerID="d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c"
Nov 01 22:48:06.012694 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.012673    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c67740c-1c80-47bd-8f03-495d6269dfbf" containerID="9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622" exitCode=0
Nov 01 22:48:06.012810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.012742    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"
Nov 01 22:48:06.012810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.012717    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2" event={"ID":"6c67740c-1c80-47bd-8f03-495d6269dfbf","Type":"ContainerDied","Data":"9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622"}
Nov 01 22:48:06.012912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.012813    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2" event={"ID":"6c67740c-1c80-47bd-8f03-495d6269dfbf","Type":"ContainerDied","Data":"d9907a1bca011cd637da95cdd34d86d00fd50ca588260e239da8786de3c1da7d"}
Nov 01 22:48:06.024075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.023954    4531 scope.go:122] "RemoveContainer" containerID="d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c"
Nov 01 22:48:06.024393 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:48:06.024313    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c\": container with ID starting with d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c not found: ID does not exist" containerID="d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c"
Nov 01 22:48:06.024393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.024352    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c"} err="failed to get container status \"d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c\": rpc error: code = NotFound desc = could not find container \"d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c\": container with ID starting with d16dbbade5007022d98485de3beb6e8aab3bba453819b2473691c5a7a9e33e4c not found: ID does not exist"
Nov 01 22:48:06.024393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.024375    4531 scope.go:122] "RemoveContainer" containerID="9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622"
Nov 01 22:48:06.036856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.036837    4531 scope.go:122] "RemoveContainer" containerID="9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622"
Nov 01 22:48:06.037382 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:48:06.037362    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622\": container with ID starting with 9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622 not found: ID does not exist" containerID="9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622"
Nov 01 22:48:06.037492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.037386    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622"} err="failed to get container status \"9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622\": rpc error: code = NotFound desc = could not find container \"9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622\": container with ID starting with 9894a9f85e19be4c7c136adb0d3ff7820fe729522700e106fe19da398f83a622 not found: ID does not exist"
Nov 01 22:48:06.050674 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.050649    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp\" (UniqueName: \"kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp\") pod \"6c67740c-1c80-47bd-8f03-495d6269dfbf\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") "
Nov 01 22:48:06.050832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.050702    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume\") pod \"6c67740c-1c80-47bd-8f03-495d6269dfbf\" (UID: \"6c67740c-1c80-47bd-8f03-495d6269dfbf\") "
Nov 01 22:48:06.051118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.051088    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume" pod "6c67740c-1c80-47bd-8f03-495d6269dfbf" (UID: "6c67740c-1c80-47bd-8f03-495d6269dfbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:06.051447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.051424    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c67740c-1c80-47bd-8f03-495d6269dfbf-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:06.051538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.051452    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsw97\" (UniqueName: \"kubernetes.io/projected/129d0479-1fbc-4f27-a779-eab25243f675-kube-api-access-rsw97\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:06.052852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.052827    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp" pod "6c67740c-1c80-47bd-8f03-495d6269dfbf" (UID: "6c67740c-1c80-47bd-8f03-495d6269dfbf"). InnerVolumeSpecName "kube-api-access-h4qfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:06.153439 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.153368    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4qfp\" (UniqueName: \"kubernetes.io/projected/6c67740c-1c80-47bd-8f03-495d6269dfbf-kube-api-access-h4qfp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:06.238108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.238078    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"]
Nov 01 22:48:06.243902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.243877    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-798f4550-787dfb7775-6rl9k"]
Nov 01 22:48:06.430347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.430274    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"]
Nov 01 22:48:06.435477 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.435454    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-798f4550-5898cb8cf5-wssl2"]
Nov 01 22:48:06.828976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.828941    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="129d0479-1fbc-4f27-a779-eab25243f675" path="/var/lib/kubelet/pods/129d0479-1fbc-4f27-a779-eab25243f675/volumes"
Nov 01 22:48:06.829570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.829549    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c67740c-1c80-47bd-8f03-495d6269dfbf" path="/var/lib/kubelet/pods/6c67740c-1c80-47bd-8f03-495d6269dfbf/volumes"
Nov 01 22:48:06.963389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:06.963354    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:10.856469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:10.856437    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:11.798961 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:11.798914    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"]
Nov 01 22:48:11.956329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:11.956297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:12.201035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:12.201001    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:12.413540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:12.413507    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:12.536331 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:48:12.536233    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:48:16.770682 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:16.770643    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:16.800330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:16.800259    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v" podUID="abee2b83-ee15-4da9-ac76-8ed6a4272b8d" containerName="payload-processing" containerID="cri-o://a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602" gracePeriod=25
Nov 01 22:48:17.081570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.081541    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:48:17.125234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.123525    4531 generic.go:356] "Generic (PLEG): container finished" podID="abee2b83-ee15-4da9-ac76-8ed6a4272b8d" containerID="a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602" exitCode=0
Nov 01 22:48:17.125234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.123826    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v" event={"ID":"abee2b83-ee15-4da9-ac76-8ed6a4272b8d","Type":"ContainerDied","Data":"a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602"}
Nov 01 22:48:17.125234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.123880    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v" event={"ID":"abee2b83-ee15-4da9-ac76-8ed6a4272b8d","Type":"ContainerDied","Data":"95ec30927d926e01e22182e50104939ee3b49804dbb72c89d333672474bfd76f"}
Nov 01 22:48:17.125234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.123906    4531 scope.go:122] "RemoveContainer" containerID="a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602"
Nov 01 22:48:17.125234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.124060    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"
Nov 01 22:48:17.136696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.136679    4531 scope.go:122] "RemoveContainer" containerID="a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602"
Nov 01 22:48:17.136983 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:48:17.136962    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602\": container with ID starting with a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602 not found: ID does not exist" containerID="a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602"
Nov 01 22:48:17.137041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.136990    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602"} err="failed to get container status \"a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602\": rpc error: code = NotFound desc = could not find container \"a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602\": container with ID starting with a7421c12549bbcc2c6d9930f1a97bfc3f67580ef000b912b4c01e8ef3ef79602 not found: ID does not exist"
Nov 01 22:48:17.218803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.218773    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume\") pod \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") "
Nov 01 22:48:17.218946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.218830    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2\" (UniqueName: \"kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2\") pod \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\" (UID: \"abee2b83-ee15-4da9-ac76-8ed6a4272b8d\") "
Nov 01 22:48:17.219144 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.219121    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume" pod "abee2b83-ee15-4da9-ac76-8ed6a4272b8d" (UID: "abee2b83-ee15-4da9-ac76-8ed6a4272b8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:48:17.219502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.219486    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:17.220900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.220874    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2" pod "abee2b83-ee15-4da9-ac76-8ed6a4272b8d" (UID: "abee2b83-ee15-4da9-ac76-8ed6a4272b8d"). InnerVolumeSpecName "kube-api-access-cprl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:48:17.321030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.320962    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cprl2\" (UniqueName: \"kubernetes.io/projected/abee2b83-ee15-4da9-ac76-8ed6a4272b8d-kube-api-access-cprl2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:48:17.556965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.556936    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"]
Nov 01 22:48:17.576194 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:17.576119    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b50b3403-85c76bf7f7-qqj9v"]
Nov 01 22:48:18.819680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:18.819644    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abee2b83-ee15-4da9-ac76-8ed6a4272b8d" path="/var/lib/kubelet/pods/abee2b83-ee15-4da9-ac76-8ed6a4272b8d/volumes"
Nov 01 22:48:21.549266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:21.549176    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:25.754391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:25.754356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:27.318985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:27.318954    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:27.538013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:27.537984    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:27.747678 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:27.747640    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:27.878165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:27.878128    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"]
Nov 01 22:48:27.925067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:27.925039    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d5014777-lqswc"]
Nov 01 22:48:28.051341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:28.051258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:28.819282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:28.819254    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7094636-6118-4728-b2b8-0fecc6d3e07d" path="/var/lib/kubelet/pods/b7094636-6118-4728-b2b8-0fecc6d3e07d/volumes"
Nov 01 22:48:28.849242 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:28.849210    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:29.105416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:29.105336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:30.068852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:30.068818    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"]
Nov 01 22:48:30.102685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:30.102650    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ab49442-9q8hs"]
Nov 01 22:48:30.820138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:30.820098    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2f14e2f-e098-4849-bbb6-d8f4e75931bd" path="/var/lib/kubelet/pods/c2f14e2f-e098-4849-bbb6-d8f4e75931bd/volumes"
Nov 01 22:48:31.180404 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:31.180318    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:31.408921 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:31.408887    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:32.396613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:32.396574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:33.995418 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:33.995380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:34.247760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:34.247666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:34.412876 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:34.412841    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"]
Nov 01 22:48:34.421427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:34.421380    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-798f4550-d2zmw"]
Nov 01 22:48:34.463661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:34.463613    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:34.818056 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:34.818015    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c89c60f-2648-4745-b189-4353a915b1f1" path="/var/lib/kubelet/pods/3c89c60f-2648-4745-b189-4353a915b1f1/volumes"
Nov 01 22:48:35.862425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:35.862393    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:36.073663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:36.073630    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:36.307442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:36.307414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:36.490411 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:36.490377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:37.249751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:37.232253    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:37.668116 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:37.668005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:39.054886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:39.054835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:39.267190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:39.267156    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:39.530517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:39.529750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:40.861065 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:40.861025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:41.156748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:41.156666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:41.377974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:41.377934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:41.379615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:41.379586    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"]
Nov 01 22:48:41.419499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:41.419438    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b50b3403-4l7fz"]
Nov 01 22:48:41.896741 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:41.896692    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:42.106201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:42.106166    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:42.532683 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:48:42.532646    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:48:42.818890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:42.818818    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6586ef74-40db-4827-8f20-94915dfe2140" path="/var/lib/kubelet/pods/6586ef74-40db-4827-8f20-94915dfe2140/volumes"
Nov 01 22:48:45.055274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:45.055240    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"]
Nov 01 22:48:45.101702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:45.101666    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a77590dd-h4n7t"]
Nov 01 22:48:45.169250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:45.169217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:45.414968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:45.414869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:46.818778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:46.818749    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7" path="/var/lib/kubelet/pods/b6e209d2-50a4-4c3f-8f4e-57c1ce47a2c7/volumes"
Nov 01 22:48:46.833417 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:46.833377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:48.229479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:48.229429    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:48.795880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:48.795841    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-tvx5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 22:48:48.796174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:48.796142    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 22:48:48.903210 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:48.903176    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:49.950209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:49.950117    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:52.115313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:52.115281    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:55.127499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:55.127456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:55.371613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:55.371579    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:55.596307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:55.596273    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:57.209267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:57.209227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:58.570217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:58.570138    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:58.824029 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:58.823937    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:48:59.432022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.431987    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"]
Nov 01 22:48:59.434113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434085    4531 cpu_manager.go:414] "containerMap: removing container" podUID="abee2b83-ee15-4da9-ac76-8ed6a4272b8d" containerName="payload-processing"
Nov 01 22:48:59.434113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434109    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abee2b83-ee15-4da9-ac76-8ed6a4272b8d" containerName="payload-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434130    4531 cpu_manager.go:414] "containerMap: removing container" podUID="129d0479-1fbc-4f27-a779-eab25243f675" containerName="payload-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434135    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="129d0479-1fbc-4f27-a779-eab25243f675" containerName="payload-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434161    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6586ef74-40db-4827-8f20-94915dfe2140" containerName="revoke-keys"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434166    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6586ef74-40db-4827-8f20-94915dfe2140" containerName="revoke-keys"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434175    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c67740c-1c80-47bd-8f03-495d6269dfbf" containerName="payload-pre-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434180    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c67740c-1c80-47bd-8f03-495d6269dfbf" containerName="payload-pre-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434300    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="abee2b83-ee15-4da9-ac76-8ed6a4272b8d" containerName="payload-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434309    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="129d0479-1fbc-4f27-a779-eab25243f675" containerName="payload-processing"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434319    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6586ef74-40db-4827-8f20-94915dfe2140" containerName="revoke-keys"
Nov 01 22:48:59.434331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.434325    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c67740c-1c80-47bd-8f03-495d6269dfbf" containerName="payload-pre-processing"
Nov 01 22:48:59.438935 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.438912    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.499018 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.498977    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"]
Nov 01 22:48:59.558773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.555896    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-945efd02\""
Nov 01 22:48:59.558773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.556026    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-945efd02-dockercfg-568fm\""
Nov 01 22:48:59.558773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.557140    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45vh6\" (UniqueName: \"kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.558773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.557214    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.659079 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.659045    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45vh6\" (UniqueName: \"kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.659566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.659122    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.660180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.660133    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.714231 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.714199    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45vh6\" (UniqueName: \"kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6\") pod \"payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:48:59.754138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:48:59.754117    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:49:00.145234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:00.145208    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"]
Nov 01 22:49:00.147227 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:49:00.147203    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01933a1d_f834_46d2_81ca_a6ae157a4800.slice/crio-a7f7e4b8d15f21e70ffd8e96f95c037e3d34e69f6340f8cbb795b1c45bac2d3d WatchSource:0}: Error finding container a7f7e4b8d15f21e70ffd8e96f95c037e3d34e69f6340f8cbb795b1c45bac2d3d: Status 404 returned error can't find the container with id a7f7e4b8d15f21e70ffd8e96f95c037e3d34e69f6340f8cbb795b1c45bac2d3d
Nov 01 22:49:00.820532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:00.820503    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:49:00.820941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:00.820558    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" event={"ID":"01933a1d-f834-46d2-81ca-a6ae157a4800","Type":"ContainerStarted","Data":"234d0bf7060d48dd64215bd48635849fbe0c73eb3317d116cc4d79c65aab74ca"}
Nov 01 22:49:00.820941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:00.820586    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" event={"ID":"01933a1d-f834-46d2-81ca-a6ae157a4800","Type":"ContainerStarted","Data":"a7f7e4b8d15f21e70ffd8e96f95c037e3d34e69f6340f8cbb795b1c45bac2d3d"}
Nov 01 22:49:00.929880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:00.929827    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" podStartSLOduration=1.929805323 podStartE2EDuration="1.929805323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:00.922827682 +0000 UTC m=+1522.240477685" watchObservedRunningTime="2026-11-01 22:49:00.929805323 +0000 UTC m=+1522.247455324"
Nov 01 22:49:01.901950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:01.901916    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"]
Nov 01 22:49:01.902427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:01.902105    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy" containerID="cri-o://fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc" gracePeriod=30
Nov 01 22:49:02.469493 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:02.469456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:05.663328 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:05.663294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:06.078148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:06.078113    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"]
Nov 01 22:49:06.124277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:06.124243    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aceb0f25-pfjfd"]
Nov 01 22:49:06.817685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:06.817650    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47901d08-5ff6-492e-94e6-47a3addec19e" path="/var/lib/kubelet/pods/47901d08-5ff6-492e-94e6-47a3addec19e/volumes"
Nov 01 22:49:07.194026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.193994    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:49:07.271468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271438    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271471    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271541    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271582    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271689 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271637    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271686    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.271877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271762    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271812    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.271877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271835    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.271877 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271867    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.272143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.271966    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.272143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.272122    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.272253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.272166    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert\") pod \"d16b3f34-867c-4036-ac6e-7151754515ad\" (UID: \"d16b3f34-867c-4036-ac6e-7151754515ad\") "
Nov 01 22:49:07.272687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.272659    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:49:07.273123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.273099    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.273227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.273128    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d16b3f34-867c-4036-ac6e-7151754515ad-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.273227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.273145    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.273227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.273161    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.273227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.273178    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.274035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.274009    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:49:07.274306 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.274288    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:49:07.274427 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.274406    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:49:07.274475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.274432    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8" pod "d16b3f34-867c-4036-ac6e-7151754515ad" (UID: "d16b3f34-867c-4036-ac6e-7151754515ad"). InnerVolumeSpecName "kube-api-access-7k7h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:49:07.384282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.384142    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7k7h8\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-kube-api-access-7k7h8\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.384282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.384176    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d16b3f34-867c-4036-ac6e-7151754515ad-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.384282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.384190    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d16b3f34-867c-4036-ac6e-7151754515ad-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.384282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.384203    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d16b3f34-867c-4036-ac6e-7151754515ad-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:49:07.431663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.431629    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:07.880125 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.880090    4531 generic.go:356] "Generic (PLEG): container finished" podID="d16b3f34-867c-4036-ac6e-7151754515ad" containerID="fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc" exitCode=0
Nov 01 22:49:07.880580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.880152    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"
Nov 01 22:49:07.880580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.880176    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" event={"ID":"d16b3f34-867c-4036-ac6e-7151754515ad","Type":"ContainerDied","Data":"fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc"}
Nov 01 22:49:07.880580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.880244    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr" event={"ID":"d16b3f34-867c-4036-ac6e-7151754515ad","Type":"ContainerDied","Data":"808ca96d5d413430a8d31302e580232879f367831ed69a79f33233d657121394"}
Nov 01 22:49:07.880580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.880272    4531 scope.go:122] "RemoveContainer" containerID="fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc"
Nov 01 22:49:07.894185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.894166    4531 scope.go:122] "RemoveContainer" containerID="fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc"
Nov 01 22:49:07.894479 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:49:07.894457    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc\": container with ID starting with fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc not found: ID does not exist" containerID="fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc"
Nov 01 22:49:07.894562 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:07.894489    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc"} err="failed to get container status \"fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc\": rpc error: code = NotFound desc = could not find container \"fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc\": container with ID starting with fe348849b00137b42d047429ad8d98c0de5489418faa91a68d1754e423f786cc not found: ID does not exist"
Nov 01 22:49:08.040021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:08.039991    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"]
Nov 01 22:49:08.085216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:08.085190    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-9d4rr"]
Nov 01 22:49:08.820331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:08.820297    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" path="/var/lib/kubelet/pods/d16b3f34-867c-4036-ac6e-7151754515ad/volumes"
Nov 01 22:49:10.051235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:10.051205    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"]
Nov 01 22:49:10.056294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:10.056260    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5108a13-kz4nm"]
Nov 01 22:49:10.819160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:10.819127    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ced4ee3-1e13-44f8-8f93-75226f143d62" path="/var/lib/kubelet/pods/5ced4ee3-1e13-44f8-8f93-75226f143d62/volumes"
Nov 01 22:49:11.829130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:11.829101    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:49:11.932128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:11.932085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:12.437029 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:12.436991    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:12.534856 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:49:12.534820    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:49:17.116614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:17.116579    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"]
Nov 01 22:49:17.154097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:17.154067    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaf3fd38-m7ll7"]
Nov 01 22:49:17.185165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:17.185137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:17.492297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:17.492266    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:17.732663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:17.732632    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:18.825979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:18.825939    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef" path="/var/lib/kubelet/pods/ad8d1ac0-5ef8-42ec-8dfc-6bac02178cef/volumes"
Nov 01 22:49:22.241766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:22.240996    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:26.055746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:26.055693    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"]
Nov 01 22:49:26.106119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:26.106079    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2e6d593-vv69z"]
Nov 01 22:49:26.851751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:26.851378    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d41d4e-078f-4809-a3c2-1a96e5fc3948" path="/var/lib/kubelet/pods/30d41d4e-078f-4809-a3c2-1a96e5fc3948/volumes"
Nov 01 22:49:26.964426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:26.964394    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:27.266752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:27.255429    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:27.510753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:27.483744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:27.726414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:27.726217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:28.304705 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:28.304668    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:31.882386 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:31.882343    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:32.122276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:32.122238    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:32.887965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:32.887933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:34.050357 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:34.050322    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"]
Nov 01 22:49:34.096533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:34.096495    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd59a338-r2cvr"]
Nov 01 22:49:34.350816 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:34.350715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:34.855852 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:34.855491    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a69837d8-c248-4181-91bf-b39cfd9791bd" path="/var/lib/kubelet/pods/a69837d8-c248-4181-91bf-b39cfd9791bd/volumes"
Nov 01 22:49:34.984859 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:34.984702    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:35.214695 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:35.214095    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:36.455229 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.455181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:36.725577 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.725492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:36.800435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.800400    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"]
Nov 01 22:49:36.801211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.801190    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy"
Nov 01 22:49:36.801211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.801208    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy"
Nov 01 22:49:36.801426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.801418    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16b3f34-867c-4036-ac6e-7151754515ad" containerName="istio-proxy"
Nov 01 22:49:36.805851 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.805831    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:36.865996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.865958    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"]
Nov 01 22:49:36.869903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.869869    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:36.870354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.870329    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z47l\" (UniqueName: \"kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l\") pod \"payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:36.906745 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.906128    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-a97162\""
Nov 01 22:49:36.906745 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.906254    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-a97162-dockercfg-d5r2r\""
Nov 01 22:49:36.959898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.959866    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"]
Nov 01 22:49:36.965850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.965827    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:36.996034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.995512    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z47l\" (UniqueName: \"kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l\") pod \"payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:36.996034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.995698    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:36.997342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:36.997304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:37.019462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.019425    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"]
Nov 01 22:49:37.053458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.053424    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z47l\" (UniqueName: \"kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l\") pod \"payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:37.097550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.097505    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js5d7\" (UniqueName: \"kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7\") pod \"payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.097799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.097593    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.122278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.122238    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:37.203533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.203221    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js5d7\" (UniqueName: \"kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7\") pod \"payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.203533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.203305    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.204403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.204373    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.310344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.310270    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js5d7\" (UniqueName: \"kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7\") pod \"payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.560593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.560502    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"]
Nov 01 22:49:37.604657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.604627    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:37.994468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:37.994435    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"]
Nov 01 22:49:37.996692 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:49:37.996663    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ba7ec5_d21e_4c2c_97b8_f278fcf4cc38.slice/crio-e308d038eb6d9fbd18a222a3e89f3081c4a1b47a854fa2adfbd8af33b1074bb4 WatchSource:0}: Error finding container e308d038eb6d9fbd18a222a3e89f3081c4a1b47a854fa2adfbd8af33b1074bb4: Status 404 returned error can't find the container with id e308d038eb6d9fbd18a222a3e89f3081c4a1b47a854fa2adfbd8af33b1074bb4
Nov 01 22:49:38.275627 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.275544    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:38.344648 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.344605    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x" event={"ID":"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38","Type":"ContainerStarted","Data":"6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db"}
Nov 01 22:49:38.344887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.344664    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x" event={"ID":"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38","Type":"ContainerStarted","Data":"e308d038eb6d9fbd18a222a3e89f3081c4a1b47a854fa2adfbd8af33b1074bb4"}
Nov 01 22:49:38.344887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.344864    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:38.347150 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.347124    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6" event={"ID":"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078","Type":"ContainerStarted","Data":"3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64"}
Nov 01 22:49:38.347272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.347164    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6" event={"ID":"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078","Type":"ContainerStarted","Data":"05e0b4a459077b836703c26696e68e41a04625bc1bde8ba4926c8b49b552c95b"}
Nov 01 22:49:38.347272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.347195    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:38.456219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.456050    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x" podStartSLOduration=2.456031793 podStartE2EDuration="2.456031793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:38.454985596 +0000 UTC m=+1559.772635597" watchObservedRunningTime="2026-11-01 22:49:38.456031793 +0000 UTC m=+1559.773681797"
Nov 01 22:49:38.569001 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:38.568912    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6" podStartSLOduration=2.568899169 podStartE2EDuration="2.568899169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:38.563877878 +0000 UTC m=+1559.881527885" watchObservedRunningTime="2026-11-01 22:49:38.568899169 +0000 UTC m=+1559.886549169"
Nov 01 22:49:39.053532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.053501    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"]
Nov 01 22:49:39.060182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.060146    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22fd8c89-62gt9"]
Nov 01 22:49:39.600960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.600929    4531 scope.go:122] "RemoveContainer" containerID="703d4c184040b2dec15fb84b605a2354e5139962ab52d8f24c1368d4d6a7ac88"
Nov 01 22:49:39.611865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.611843    4531 scope.go:122] "RemoveContainer" containerID="eddee2ebfe5a26d8be0205fb296ffbfa69cadf3be8fc6538069be65583405a53"
Nov 01 22:49:39.624660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.624637    4531 scope.go:122] "RemoveContainer" containerID="79c0bfac5147fd36295a48c78446887d41d1e6c725f5da240ff80144634833e5"
Nov 01 22:49:39.714371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.714347    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:39.972039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:39.972006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:40.819923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:40.819893    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea68789-157d-477e-acda-10b572466456" path="/var/lib/kubelet/pods/3ea68789-157d-477e-acda-10b572466456/volumes"
Nov 01 22:49:42.526615 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:49:42.526580    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:49:43.102714 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.102683    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"]
Nov 01 22:49:43.152049 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.151954    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d004d649-92bh9"]
Nov 01 22:49:43.198812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.192652    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:43.452699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.452641    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:43.500923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.500891    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"]
Nov 01 22:49:43.508697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.508669    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.581751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.570933    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"]
Nov 01 22:49:43.581751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.571924    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-de3d6ade-dockercfg-5zvdq\""
Nov 01 22:49:43.581751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.572036    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-de3d6ade\""
Nov 01 22:49:43.617785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.605450    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"]
Nov 01 22:49:43.617785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.607183    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.617785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.607548    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nds5q\" (UniqueName: \"kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q\") pod \"payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.617785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.614071    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:43.799201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.780002    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nds5q\" (UniqueName: \"kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q\") pod \"payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.799201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.780152    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.799201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.780211    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume\") pod \"payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") " pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:43.799201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.781405    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.799201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.780251    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8vgj\" (UniqueName: \"kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj\") pod \"payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") " pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:43.803414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.803383    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:43.817139 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.815766    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"]
Nov 01 22:49:43.872287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.872254    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nds5q\" (UniqueName: \"kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q\") pod \"payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:43.885711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.885664    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume\") pod \"payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") " pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:43.885939 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.885718    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8vgj\" (UniqueName: \"kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj\") pod \"payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") " pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:43.886981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.886921    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume\") pod \"payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") " pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:43.940857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:43.940820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8vgj\" (UniqueName: \"kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj\") pod \"payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") " pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:44.118533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:44.118431    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:44.128463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:44.128435    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:44.617292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:44.617260    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"]
Nov 01 22:49:44.619880 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:49:44.619849    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a40114_fc1a_4389_a7b3_999bbd0882f0.slice/crio-dcd6d77f57667390ea779dfe1268de4d0c52ad10a3df3e8356e11c2690679f7d WatchSource:0}: Error finding container dcd6d77f57667390ea779dfe1268de4d0c52ad10a3df3e8356e11c2690679f7d: Status 404 returned error can't find the container with id dcd6d77f57667390ea779dfe1268de4d0c52ad10a3df3e8356e11c2690679f7d
Nov 01 22:49:44.676293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:44.676270    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"]
Nov 01 22:49:44.677866 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:49:44.677838    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68af80e8_66f0_4d77_903b_f12601fd3fcc.slice/crio-37ac54536da255c12b59309adacfc1bb5a40e4d52e1a9febda67c23bfea652f1 WatchSource:0}: Error finding container 37ac54536da255c12b59309adacfc1bb5a40e4d52e1a9febda67c23bfea652f1: Status 404 returned error can't find the container with id 37ac54536da255c12b59309adacfc1bb5a40e4d52e1a9febda67c23bfea652f1
Nov 01 22:49:44.820826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:44.820790    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7ab4e74-c70b-4dd5-823f-d208cf8fb02e" path="/var/lib/kubelet/pods/c7ab4e74-c70b-4dd5-823f-d208cf8fb02e/volumes"
Nov 01 22:49:45.416999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.416957    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq" event={"ID":"b2a40114-fc1a-4389-a7b3-999bbd0882f0","Type":"ContainerStarted","Data":"2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b"}
Nov 01 22:49:45.417191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.417025    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq" event={"ID":"b2a40114-fc1a-4389-a7b3-999bbd0882f0","Type":"ContainerStarted","Data":"dcd6d77f57667390ea779dfe1268de4d0c52ad10a3df3e8356e11c2690679f7d"}
Nov 01 22:49:45.417191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.417080    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:45.418945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.418912    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn" event={"ID":"68af80e8-66f0-4d77-903b-f12601fd3fcc","Type":"ContainerStarted","Data":"b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda"}
Nov 01 22:49:45.419105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.418974    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn" event={"ID":"68af80e8-66f0-4d77-903b-f12601fd3fcc","Type":"ContainerStarted","Data":"37ac54536da255c12b59309adacfc1bb5a40e4d52e1a9febda67c23bfea652f1"}
Nov 01 22:49:45.419105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.418997    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:45.638679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.638616    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn" podStartSLOduration=2.638596757 podStartE2EDuration="2.638596757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:45.635075182 +0000 UTC m=+1566.952725184" watchObservedRunningTime="2026-11-01 22:49:45.638596757 +0000 UTC m=+1566.956246761"
Nov 01 22:49:45.639330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:45.639286    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq" podStartSLOduration=2.639275235 podStartE2EDuration="2.639275235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:49:45.5276678 +0000 UTC m=+1566.845317806" watchObservedRunningTime="2026-11-01 22:49:45.639275235 +0000 UTC m=+1566.956925238"
Nov 01 22:49:49.129653 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:49.129615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:49.359038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:49.359005    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 22:49:49.361248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:49.361225    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 22:49:49.363305 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:49.363277    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:52.446339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:52.446301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:52.641668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:52.641626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:54.238888 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:54.238762    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:49:56.430866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:56.430836    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:49:56.431285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:56.430925    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:49:59.221599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:49:59.221554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:04.053349 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:04.053312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:06.069432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:06.069395    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"]
Nov 01 22:50:06.081242 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:06.081211    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57144b1c-8rqbj"]
Nov 01 22:50:06.819191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:06.819160    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0af721f-fd65-4794-9c71-29b6cb7aa627" path="/var/lib/kubelet/pods/a0af721f-fd65-4794-9c71-29b6cb7aa627/volumes"
Nov 01 22:50:08.923639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:08.923605    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:11.049135 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:11.049103    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"]
Nov 01 22:50:11.095171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:11.095144    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c54f5e5e-rjmzf"]
Nov 01 22:50:12.527481 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:50:12.527443    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:50:12.818464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:12.818387    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea3d6ba-85c4-443c-ad1f-d25de767e381" path="/var/lib/kubelet/pods/dea3d6ba-85c4-443c-ad1f-d25de767e381/volumes"
Nov 01 22:50:13.750380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:13.750160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:17.069964 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:17.069930    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"]
Nov 01 22:50:17.113424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:17.113388    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24c9ef6c-lm82c"]
Nov 01 22:50:18.249367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:18.249312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:18.819976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:18.819944    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad9c4429-dac1-4fae-9132-a815c3e3718f" path="/var/lib/kubelet/pods/ad9c4429-dac1-4fae-9132-a815c3e3718f/volumes"
Nov 01 22:50:22.048371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.048336    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"]
Nov 01 22:50:22.094770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.094719    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9172dfd-w2kcp"]
Nov 01 22:50:22.554932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.554899    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"]
Nov 01 22:50:22.561102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.561077    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.612021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.611987    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:50:22.612190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.612050    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:50:22.618453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.618421    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"]
Nov 01 22:50:22.748239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.748204    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.748434 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.748355    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzgw8\" (UniqueName: \"kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.748508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.748486    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.820588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.820507    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b512c2-859d-4ef1-9327-0fa75c5ba9c8" path="/var/lib/kubelet/pods/f7b512c2-859d-4ef1-9327-0fa75c5ba9c8/volumes"
Nov 01 22:50:22.850260 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.850238    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.850387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.850329    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzgw8\" (UniqueName: \"kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.850429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.850398    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.851010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.850986    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.852502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.852485    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:22.905999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:22.905970    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzgw8\" (UniqueName: \"kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8\") pod \"maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:23.179230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:23.179155    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:23.565436 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:50:23.565406    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccb6d97d_0c9a_47ee_9f2c_b5deeaea82df.slice/crio-289f5e8caa9bb7c17207ce9a5c8bf5fc012b5d22f7ad7c85d97c96a3bc44a221 WatchSource:0}: Error finding container 289f5e8caa9bb7c17207ce9a5c8bf5fc012b5d22f7ad7c85d97c96a3bc44a221: Status 404 returned error can't find the container with id 289f5e8caa9bb7c17207ce9a5c8bf5fc012b5d22f7ad7c85d97c96a3bc44a221
Nov 01 22:50:23.568443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:23.568414    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"]
Nov 01 22:50:23.633769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:23.633711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:23.810789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:23.810755    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv" event={"ID":"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df","Type":"ContainerStarted","Data":"289f5e8caa9bb7c17207ce9a5c8bf5fc012b5d22f7ad7c85d97c96a3bc44a221"}
Nov 01 22:50:24.048537 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:24.048508    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"]
Nov 01 22:50:24.071524 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:24.071495    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ae58dcd-dt4wp"]
Nov 01 22:50:24.830797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:24.830764    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ee907ae-6811-438b-8ea2-73aca9aca133" path="/var/lib/kubelet/pods/8ee907ae-6811-438b-8ea2-73aca9aca133/volumes"
Nov 01 22:50:24.971211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:24.971177    4531 generic.go:356] "Generic (PLEG): container finished" podID="ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" containerID="7ee54e58137001b084128c019054a72eaa805445560ef0039c7d7521e135d802" exitCode=0
Nov 01 22:50:24.971849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:24.971816    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv" event={"ID":"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df","Type":"ContainerDied","Data":"7ee54e58137001b084128c019054a72eaa805445560ef0039c7d7521e135d802"}
Nov 01 22:50:26.153261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.153238    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:26.264374 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.264269    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8\" (UniqueName: \"kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8\") pod \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") "
Nov 01 22:50:26.265314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.264611    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca\") pod \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") "
Nov 01 22:50:26.265314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.264791    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca\") pod \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\" (UID: \"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df\") "
Nov 01 22:50:26.265314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.264966    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca" pod "ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" (UID: "ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:50:26.265707 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.265577    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:26.267827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.267786    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca" pod "ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" (UID: "ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:50:26.268323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.268302    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8" pod "ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" (UID: "ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df"). InnerVolumeSpecName "kube-api-access-lzgw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:50:26.367311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.367282    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:26.367311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.367308    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzgw8\" (UniqueName: \"kubernetes.io/projected/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df-kube-api-access-lzgw8\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:26.991117 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.991090    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"
Nov 01 22:50:26.991285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.991130    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv" event={"ID":"ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df","Type":"ContainerDied","Data":"289f5e8caa9bb7c17207ce9a5c8bf5fc012b5d22f7ad7c85d97c96a3bc44a221"}
Nov 01 22:50:26.991285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:26.991174    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="289f5e8caa9bb7c17207ce9a5c8bf5fc012b5d22f7ad7c85d97c96a3bc44a221"
Nov 01 22:50:29.122295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:29.122258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:29.339698 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:29.339653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:29.573711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:29.573669    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:30.285176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:30.285139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:30.486341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:30.486304    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:34.286886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.286847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:34.496542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.496509    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:34.731901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.731865    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"]
Nov 01 22:50:34.732856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.732833    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" containerName="revoke-keys"
Nov 01 22:50:34.732856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.732854    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" containerName="revoke-keys"
Nov 01 22:50:34.733075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.733058    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" containerName="revoke-keys"
Nov 01 22:50:34.737765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.737743    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.795167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.795129    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"]
Nov 01 22:50:34.828344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.828147    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.828645 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.828608    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.829450 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.828956    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4pk\" (UniqueName: \"kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.832422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.832397    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:50:34.832422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.832406    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:50:34.930151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.930120    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4pk\" (UniqueName: \"kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.930285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.930226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.930285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.930260    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.931045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.931023    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.932836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.932817    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:34.986795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:34.986712    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4pk\" (UniqueName: \"kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk\") pod \"maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:35.051431 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:35.051409    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"]
Nov 01 22:50:35.053453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:35.053433    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:35.056186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:35.056162    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-16899b33-r6998"]
Nov 01 22:50:35.491837 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:35.487975    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"]
Nov 01 22:50:36.081161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.080881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:36.131201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.131161    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4" event={"ID":"6f1865e2-b0af-40d6-b3de-3cbd1d47737e","Type":"ContainerStarted","Data":"d3dc8da8cd8660c619a191e1c481c2bdc97aa8302e1de62b0794e2710f2382f1"}
Nov 01 22:50:36.131522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.131501    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4" event={"ID":"6f1865e2-b0af-40d6-b3de-3cbd1d47737e","Type":"ContainerStarted","Data":"fabd7eb5266170c8bf7bdf83150ecd938c5c956ee301fd1014d9444ec56f45eb"}
Nov 01 22:50:36.254802 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.254746    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4" podStartSLOduration=2.254716738 podStartE2EDuration="2.254716738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:50:36.250882493 +0000 UTC m=+1617.568532495" watchObservedRunningTime="2026-11-01 22:50:36.254716738 +0000 UTC m=+1617.572366786"
Nov 01 22:50:36.348339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.348258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:36.673600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.673574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:36.819722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:36.819695    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203" path="/var/lib/kubelet/pods/b40cd1a4-a9c4-4fca-bea7-7d4c23f0a203/volumes"
Nov 01 22:50:37.141179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:37.141147    4531 generic.go:356] "Generic (PLEG): container finished" podID="6f1865e2-b0af-40d6-b3de-3cbd1d47737e" containerID="d3dc8da8cd8660c619a191e1c481c2bdc97aa8302e1de62b0794e2710f2382f1" exitCode=0
Nov 01 22:50:37.141346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:37.141231    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4" event={"ID":"6f1865e2-b0af-40d6-b3de-3cbd1d47737e","Type":"ContainerDied","Data":"d3dc8da8cd8660c619a191e1c481c2bdc97aa8302e1de62b0794e2710f2382f1"}
Nov 01 22:50:38.329573 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.329549    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:38.386629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.386573    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca\") pod \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") "
Nov 01 22:50:38.386804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.386690    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk\" (UniqueName: \"kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk\") pod \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") "
Nov 01 22:50:38.386895 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.386881    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca\") pod \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\" (UID: \"6f1865e2-b0af-40d6-b3de-3cbd1d47737e\") "
Nov 01 22:50:38.387179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.387156    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca" pod "6f1865e2-b0af-40d6-b3de-3cbd1d47737e" (UID: "6f1865e2-b0af-40d6-b3de-3cbd1d47737e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:50:38.387497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.387481    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:38.388765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.388709    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca" pod "6f1865e2-b0af-40d6-b3de-3cbd1d47737e" (UID: "6f1865e2-b0af-40d6-b3de-3cbd1d47737e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:50:38.388862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.388786    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk" pod "6f1865e2-b0af-40d6-b3de-3cbd1d47737e" (UID: "6f1865e2-b0af-40d6-b3de-3cbd1d47737e"). InnerVolumeSpecName "kube-api-access-zl4pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:50:38.489765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.489690    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:38.489765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:38.489744    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl4pk\" (UniqueName: \"kubernetes.io/projected/6f1865e2-b0af-40d6-b3de-3cbd1d47737e-kube-api-access-zl4pk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:50:39.161204 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.161132    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"
Nov 01 22:50:39.161348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.161218    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4" event={"ID":"6f1865e2-b0af-40d6-b3de-3cbd1d47737e","Type":"ContainerDied","Data":"fabd7eb5266170c8bf7bdf83150ecd938c5c956ee301fd1014d9444ec56f45eb"}
Nov 01 22:50:39.161348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.161268    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fabd7eb5266170c8bf7bdf83150ecd938c5c956ee301fd1014d9444ec56f45eb"
Nov 01 22:50:39.675191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.675158    4531 scope.go:122] "RemoveContainer" containerID="dcf076963487b0349faed467ebe7a2fc6ede330f63396bae9d58b6e139db3a88"
Nov 01 22:50:39.686479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.686458    4531 scope.go:122] "RemoveContainer" containerID="2864d9f90904ee8e49f96082ff0702ff55d1bde6322692e255c9fc3a07574e8b"
Nov 01 22:50:39.696690 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.696669    4531 scope.go:122] "RemoveContainer" containerID="ac14cef12d07c4e5a31933c7a8a9b65d35b84456c6e2d17b7e8243877e691adf"
Nov 01 22:50:39.709299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.709277    4531 scope.go:122] "RemoveContainer" containerID="ac46db010d04a51727bbdf077bfa0d83cad117e3f314b1f6ac959b9e17deffd2"
Nov 01 22:50:39.719037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.719020    4531 scope.go:122] "RemoveContainer" containerID="0f335d3c0fa82caa710e4cbe8b08b6570bbe2346ba0d3ea10b84c8d551960ab1"
Nov 01 22:50:39.728628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.728611    4531 scope.go:122] "RemoveContainer" containerID="8c8e746de7a4d03543094db93b88b723a73ac361ff4cd400c033e5d7d258bbb8"
Nov 01 22:50:39.740089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.740060    4531 scope.go:122] "RemoveContainer" containerID="e9f6efc3a3da9b15846c8d82e376fb9a90b38627df47d6190342149f69ec9365"
Nov 01 22:50:39.749898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:39.749881    4531 scope.go:122] "RemoveContainer" containerID="7bbbbbe1dfbb15af1e1f76aac33a8c0fe92b414f083d9761aaa8df4bd7be71ae"
Nov 01 22:50:40.078611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:40.078577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:40.313814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:40.313773    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:42.529688 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:50:42.529650    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:50:45.067976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:45.067914    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:49.793967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:49.793936    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:51.820605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:51.820550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:52.078829 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:52.078699    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:52.644697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:52.644638    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:54.669277 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:54.669237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:58.631038 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:58.631006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:50:59.545095 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:50:59.545053    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:02.622266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:02.622217    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:04.848215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:04.848170    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:06.725034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:06.724997    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:07.303244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:07.303208    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:09.711201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:09.711156    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:12.532269 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:51:12.532232    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:51:13.241328 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:13.241289    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:14.512715 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:14.512671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:18.605574 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:18.605539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:20.106105 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.106022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:20.315996 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.315957    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:20.602414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.602379    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:20.720482 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.720453    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"]
Nov 01 22:51:20.776785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.776752    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d04c71aa-9mvqv"]
Nov 01 22:51:20.821339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.821300    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df" path="/var/lib/kubelet/pods/ccb6d97d-0c9a-47ee-9f2c-b5deeaea82df/volumes"
Nov 01 22:51:20.864615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:20.864542    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:21.620335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:21.620302    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:22.247005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:22.244106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:22.480452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:22.480419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:24.948014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:24.947975    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:26.582064 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:26.582031    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:26.852855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:26.852714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:27.024531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:27.024493    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"]
Nov 01 22:51:27.040088 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:27.040011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:27.070902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:27.070867    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-68a7f708-rt4r4"]
Nov 01 22:51:27.234765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:27.234714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:27.553522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:27.553394    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:27.779772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:27.779721    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:28.109206 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:28.109173    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:28.594385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:28.594324    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:28.866364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:28.866299    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f1865e2-b0af-40d6-b3de-3cbd1d47737e" path="/var/lib/kubelet/pods/6f1865e2-b0af-40d6-b3de-3cbd1d47737e/volumes"
Nov 01 22:51:28.888903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:28.888861    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:30.806334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:30.806297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:31.019801 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.019141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:31.325474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.325442    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"]
Nov 01 22:51:31.326389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.326365    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f1865e2-b0af-40d6-b3de-3cbd1d47737e" containerName="revoke-keys"
Nov 01 22:51:31.326389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.326387    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f1865e2-b0af-40d6-b3de-3cbd1d47737e" containerName="revoke-keys"
Nov 01 22:51:31.326573 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.326564    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f1865e2-b0af-40d6-b3de-3cbd1d47737e" containerName="revoke-keys"
Nov 01 22:51:31.330243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.330218    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:31.399344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.399262    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"]
Nov 01 22:51:31.428092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.428061    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-646881\""
Nov 01 22:51:31.428297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.428074    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-646881-dockercfg-7tl8c\""
Nov 01 22:51:31.515193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.515157    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:31.515665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.515642    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr6df\" (UniqueName: \"kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df\") pod \"payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:31.536046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.536017    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"]
Nov 01 22:51:31.550772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.550383    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:31.623089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.622989    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"]
Nov 01 22:51:31.665716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.665678    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr6df\" (UniqueName: \"kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df\") pod \"payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:31.666600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.666544    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:31.672809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.667804    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:31.769232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.768403    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:31.769232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.768526    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc5x4\" (UniqueName: \"kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4\") pod \"payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:31.884805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.869910    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:31.884805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.870045    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc5x4\" (UniqueName: \"kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4\") pod \"payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:31.884805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.871701    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:31.988806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:31.988766    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc5x4\" (UniqueName: \"kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4\") pod \"payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:32.000986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.000945    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr6df\" (UniqueName: \"kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df\") pod \"payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:32.210228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.210190    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:32.253758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.252984    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:32.645082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.645051    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"]
Nov 01 22:51:32.748427 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:51:32.748383    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b5e9e9_47f1_42b8_ad1b_a237035b00e0.slice/crio-c598bbc0b872ea83f1ace179ba80fcbf4b4cb855e7f06d58be6adc360ef25eda WatchSource:0}: Error finding container c598bbc0b872ea83f1ace179ba80fcbf4b4cb855e7f06d58be6adc360ef25eda: Status 404 returned error can't find the container with id c598bbc0b872ea83f1ace179ba80fcbf4b4cb855e7f06d58be6adc360ef25eda
Nov 01 22:51:32.750177 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.750151    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"]
Nov 01 22:51:32.870756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.869577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:32.994757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.991900    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf" event={"ID":"29b5e9e9-47f1-42b8-ad1b-a237035b00e0","Type":"ContainerStarted","Data":"9a2ee73f7dbcfc79403c6883c9b2d0e28a337101453f97c603d23dd0ee651be5"}
Nov 01 22:51:32.994757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.991966    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf" event={"ID":"29b5e9e9-47f1-42b8-ad1b-a237035b00e0","Type":"ContainerStarted","Data":"c598bbc0b872ea83f1ace179ba80fcbf4b4cb855e7f06d58be6adc360ef25eda"}
Nov 01 22:51:32.994757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.993136    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:33.008789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.995596    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf" event={"ID":"2d66bf74-6646-47bc-8291-a39d93c4f047","Type":"ContainerStarted","Data":"a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0"}
Nov 01 22:51:33.008789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.995644    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf" event={"ID":"2d66bf74-6646-47bc-8291-a39d93c4f047","Type":"ContainerStarted","Data":"1b15b25e62e4a3b77ff61b36d8ae270c4f8830a87e6621f1f7c69071db74d6dd"}
Nov 01 22:51:33.008789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:32.996431    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:33.090602 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:33.079240    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:33.102723 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:33.102670    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf" podStartSLOduration=2.102655722 podStartE2EDuration="2.102655722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:33.102054427 +0000 UTC m=+1674.419704428" watchObservedRunningTime="2026-11-01 22:51:33.102655722 +0000 UTC m=+1674.420305722"
Nov 01 22:51:33.218217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:33.218164    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf" podStartSLOduration=2.218149386 podStartE2EDuration="2.218149386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:51:33.214857607 +0000 UTC m=+1674.532507609" watchObservedRunningTime="2026-11-01 22:51:33.218149386 +0000 UTC m=+1674.535799419"
Nov 01 22:51:33.307794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:33.307680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:34.287864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:34.287821    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:34.695067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:34.695029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:35.564958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:35.513328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:35.870282 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:35.870183    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:37.025613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:37.025581    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:38.383585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:38.383547    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:39.825009 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:39.824982    4531 scope.go:122] "RemoveContainer" containerID="c5ef009573b33d2de4ebeb5a9f26e4626a0a40f40de2c3dd71080eb1c0dd9329"
Nov 01 22:51:39.836718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:39.836695    4531 scope.go:122] "RemoveContainer" containerID="f9220a0d13e40990b4292d9310876cd4731ecc1d6690be465b2db92886eee647"
Nov 01 22:51:39.846752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:39.846719    4531 scope.go:122] "RemoveContainer" containerID="2a0dfea68aedc8c78af60e5d9a75cecae7d52383c332620f854bb4270acdffde"
Nov 01 22:51:39.856764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:39.856747    4531 scope.go:122] "RemoveContainer" containerID="41f2abbaa359b2f13f8649289b0d629df9ad444ffd4979d9ea4044970d60b4b1"
Nov 01 22:51:39.866288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:39.866265    4531 scope.go:122] "RemoveContainer" containerID="77497ca59893a920acf24a2b4e985987045075ceb948aa4f8603fde79884ebd9"
Nov 01 22:51:39.876874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:39.876853    4531 scope.go:122] "RemoveContainer" containerID="807fc375e5ab821e240834f98a6d091e01c7852b3f308bce7d175e331dd2a638"
Nov 01 22:51:40.160992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:40.160898    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:40.422950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:40.422870    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:41.292368 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:41.292338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:41.481644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:41.481578    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:42.528048 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:51:42.528010    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:51:43.704507 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:43.704470    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:44.890129 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:44.890092    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"]
Nov 01 22:51:44.890613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:44.890321    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" containerID="cri-o://38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c" gracePeriod=30
Nov 01 22:51:45.020270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:45.020242    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 22:51:45.020451 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:45.020317    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 22:51:45.034297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:45.034268    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"]
Nov 01 22:51:45.034512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:45.034487    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy" containerID="cri-o://f968c79073ef885e3659936d700c30c299d036f6d04d7b9a5f56732f7dfb3bd2" gracePeriod=30
Nov 01 22:51:47.206046 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:47.206005    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:47.842976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:47.842935    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:48.056892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:48.056860    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:48.384253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:48.384220    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:48.844902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:48.844869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:49.048301 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:49.048251    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:49.234516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:49.234481    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:49.616762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:49.616658    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:51:49.617316 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:49.617286    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:51:50.194027 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.193944    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:51:50.209529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.209371    4531 generic.go:356] "Generic (PLEG): container finished" podID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerID="38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c" exitCode=0
Nov 01 22:51:50.209529 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.209414    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"
Nov 01 22:51:50.209702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.209604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" event={"ID":"3639a8b0-f32f-464b-8917-a3d5656f56a2","Type":"ContainerDied","Data":"38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c"}
Nov 01 22:51:50.209702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.209659    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv" event={"ID":"3639a8b0-f32f-464b-8917-a3d5656f56a2","Type":"ContainerDied","Data":"96ae85161d933ed1094039b891721734d71665733dcda1cdd54a7bbe51f11318"}
Nov 01 22:51:50.209702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.209689    4531 scope.go:122] "RemoveContainer" containerID="38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c"
Nov 01 22:51:50.211912 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.211894    4531 generic.go:356] "Generic (PLEG): container finished" podID="14d4724b-596f-4e61-9c42-d187d356ea08" containerID="f968c79073ef885e3659936d700c30c299d036f6d04d7b9a5f56732f7dfb3bd2" exitCode=0
Nov 01 22:51:50.212011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.211972    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" event={"ID":"14d4724b-596f-4e61-9c42-d187d356ea08","Type":"ContainerDied","Data":"f968c79073ef885e3659936d700c30c299d036f6d04d7b9a5f56732f7dfb3bd2"}
Nov 01 22:51:50.224978 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.224960    4531 scope.go:122] "RemoveContainer" containerID="38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c"
Nov 01 22:51:50.225241 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:51:50.225215    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c\": container with ID starting with 38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c not found: ID does not exist" containerID="38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c"
Nov 01 22:51:50.225328 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.225248    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c"} err="failed to get container status \"38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c\": rpc error: code = NotFound desc = could not find container \"38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c\": container with ID starting with 38a5e2fac3dc63a3657383ea855b7b37878b3a2f6b2aece6cbdfd0591c02e85c not found: ID does not exist"
Nov 01 22:51:50.319567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.319538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.319721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.319611    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.319721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.319696    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.319845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.319823    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.319934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.319894    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.320006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.319941    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.320067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320002    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.320067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320003    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.320174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320086    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.320174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320160    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.320293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320227    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data\") pod \"3639a8b0-f32f-464b-8917-a3d5656f56a2\" (UID: \"3639a8b0-f32f-464b-8917-a3d5656f56a2\") "
Nov 01 22:51:50.320293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320240    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.320442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320422    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.320600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320566    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.320930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320912    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.321026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320933    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.321026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320947    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.321026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320956    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3639a8b0-f32f-464b-8917-a3d5656f56a2-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.321026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.320965    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.322666 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.322642    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.322787 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.322705    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.323011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.322982    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "kube-api-access-s2x6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.323348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.323320    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo" pod "3639a8b0-f32f-464b-8917-a3d5656f56a2" (UID: "3639a8b0-f32f-464b-8917-a3d5656f56a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.346066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.346047    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:51:50.421549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.421514    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.421694 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.421653    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.421813 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.421795    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.421900 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.421884    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.421982    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422038    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422091    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422136    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422164    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422178    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422178    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket\") pod \"14d4724b-596f-4e61-9c42-d187d356ea08\" (UID: \"14d4724b-596f-4e61-9c42-d187d356ea08\") "
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422283    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.422496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422465    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.423022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.422637    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.423078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423036    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423078 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423061    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2x6s\" (UniqueName: \"kubernetes.io/projected/3639a8b0-f32f-464b-8917-a3d5656f56a2-kube-api-access-s2x6s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423077    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14d4724b-596f-4e61-9c42-d187d356ea08-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423091    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423106    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423120    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423133    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3639a8b0-f32f-464b-8917-a3d5656f56a2-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423146    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.423184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.423160    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.424340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.424320    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.424607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.424590    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:51:50.424917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.424894    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:51:50.426148 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.426122    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw" pod "14d4724b-596f-4e61-9c42-d187d356ea08" (UID: "14d4724b-596f-4e61-9c42-d187d356ea08"). InnerVolumeSpecName "kube-api-access-c77rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:50.524606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.524578    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14d4724b-596f-4e61-9c42-d187d356ea08-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.524792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.524608    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c77rw\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-kube-api-access-c77rw\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.524792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.524622    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14d4724b-596f-4e61-9c42-d187d356ea08-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.524792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.524634    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14d4724b-596f-4e61-9c42-d187d356ea08-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:50.630006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.629969    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"]
Nov 01 22:51:50.634394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.634370    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-xcjnv"]
Nov 01 22:51:50.821763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:50.821674    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" path="/var/lib/kubelet/pods/3639a8b0-f32f-464b-8917-a3d5656f56a2/volumes"
Nov 01 22:51:51.222059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:51.222031    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"
Nov 01 22:51:51.222059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:51.222042    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d" event={"ID":"14d4724b-596f-4e61-9c42-d187d356ea08","Type":"ContainerDied","Data":"19641fd2026f863cd09e6dd3cea6cd4d2ef66c2b6597bc040cafb1045dd9b8c6"}
Nov 01 22:51:51.222299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:51.222107    4531 scope.go:122] "RemoveContainer" containerID="f968c79073ef885e3659936d700c30c299d036f6d04d7b9a5f56732f7dfb3bd2"
Nov 01 22:51:51.380392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:51.380360    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"]
Nov 01 22:51:51.426378 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:51.426350    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-t785d"]
Nov 01 22:51:52.819824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:52.819786    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" path="/var/lib/kubelet/pods/14d4724b-596f-4e61-9c42-d187d356ea08/volumes"
Nov 01 22:51:54.384794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:54.384763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:54.591585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:54.591542    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:51:55.315348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.315315    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"]
Nov 01 22:51:55.316140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.316123    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy"
Nov 01 22:51:55.316140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.316140    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy"
Nov 01 22:51:55.316247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.316151    4531 cpu_manager.go:414] "containerMap: removing container" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy"
Nov 01 22:51:55.316247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.316157    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy"
Nov 01 22:51:55.316309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.316296    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3639a8b0-f32f-464b-8917-a3d5656f56a2" containerName="istio-proxy"
Nov 01 22:51:55.316309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.316305    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="14d4724b-596f-4e61-9c42-d187d356ea08" containerName="istio-proxy"
Nov 01 22:51:55.321616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.321598    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.373673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.373646    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"]
Nov 01 22:51:55.399302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.399273    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.399613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.399493    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8pp6\" (UniqueName: \"kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.399613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.399564    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.413855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.413805    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:51:55.414033 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.414014    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:51:55.501430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.501400    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8pp6\" (UniqueName: \"kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.501558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.501494    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.501637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.501596    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.505754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.503109    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.517437 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.517405    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.555706 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.555671    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8pp6\" (UniqueName: \"kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6\") pod \"maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:55.636163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:55.636106    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:56.015433 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:56.015360    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"]
Nov 01 22:51:56.018264 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:51:56.018232    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd49de9_75ff_4d35_b3a4_7c47dc0fb445.slice/crio-7b945eab086eb37211b3a5bbf00872e72277072b3af01a12730bd9a94f56b56f WatchSource:0}: Error finding container 7b945eab086eb37211b3a5bbf00872e72277072b3af01a12730bd9a94f56b56f: Status 404 returned error can't find the container with id 7b945eab086eb37211b3a5bbf00872e72277072b3af01a12730bd9a94f56b56f
Nov 01 22:51:56.275660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:56.275635    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m" event={"ID":"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445","Type":"ContainerStarted","Data":"7b945eab086eb37211b3a5bbf00872e72277072b3af01a12730bd9a94f56b56f"}
Nov 01 22:51:57.285484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:57.285450    4531 generic.go:356] "Generic (PLEG): container finished" podID="4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" containerID="7710dad263fbc9ff2f0871f3599b5a0b0b68541bee65eccf71db0c2e02940dec" exitCode=0
Nov 01 22:51:57.285898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:57.285538    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m" event={"ID":"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445","Type":"ContainerDied","Data":"7710dad263fbc9ff2f0871f3599b5a0b0b68541bee65eccf71db0c2e02940dec"}
Nov 01 22:51:58.456836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.456801    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:58.540414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.540387    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca\") pod \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") "
Nov 01 22:51:58.540596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.540583    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca\") pod \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") "
Nov 01 22:51:58.540684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.540672    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6\" (UniqueName: \"kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6\") pod \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\" (UID: \"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445\") "
Nov 01 22:51:58.540886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.540858    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca" pod "4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" (UID: "4fd49de9-75ff-4d35-b3a4-7c47dc0fb445"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:51:58.541270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.541247    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:58.542660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.542643    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6" pod "4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" (UID: "4fd49de9-75ff-4d35-b3a4-7c47dc0fb445"). InnerVolumeSpecName "kube-api-access-f8pp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:51:58.542920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.542891    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca" pod "4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" (UID: "4fd49de9-75ff-4d35-b3a4-7c47dc0fb445"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:51:58.642845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.642812    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:58.643000 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:58.642849    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8pp6\" (UniqueName: \"kubernetes.io/projected/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445-kube-api-access-f8pp6\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:51:59.304901 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:59.304872    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"
Nov 01 22:51:59.305068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:59.304911    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m" event={"ID":"4fd49de9-75ff-4d35-b3a4-7c47dc0fb445","Type":"ContainerDied","Data":"7b945eab086eb37211b3a5bbf00872e72277072b3af01a12730bd9a94f56b56f"}
Nov 01 22:51:59.305068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:59.305002    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b945eab086eb37211b3a5bbf00872e72277072b3af01a12730bd9a94f56b56f"
Nov 01 22:51:59.628275 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:51:59.628204    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:04.647205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:04.647116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:06.941155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:06.941120    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"]
Nov 01 22:52:07.056273 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:07.056235    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"]
Nov 01 22:52:07.405803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:07.405755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:07.603230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:07.603194    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:09.458075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:09.458039    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:09.692622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:09.692569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:11.942070 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:11.942034    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn" podUID="68af80e8-66f0-4d77-903b-f12601fd3fcc" containerName="payload-processing" containerID="cri-o://b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda" gracePeriod=25
Nov 01 22:52:12.056911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.056879    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq" podUID="b2a40114-fc1a-4389-a7b3-999bbd0882f0" containerName="payload-pre-processing" containerID="cri-o://2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b" gracePeriod=25
Nov 01 22:52:12.268822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.268782    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:52:12.358140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.358118    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:52:12.373684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.373660    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume\") pod \"68af80e8-66f0-4d77-903b-f12601fd3fcc\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") "
Nov 01 22:52:12.373882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.373860    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj\" (UniqueName: \"kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj\") pod \"68af80e8-66f0-4d77-903b-f12601fd3fcc\" (UID: \"68af80e8-66f0-4d77-903b-f12601fd3fcc\") "
Nov 01 22:52:12.374091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.374050    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume" pod "68af80e8-66f0-4d77-903b-f12601fd3fcc" (UID: "68af80e8-66f0-4d77-903b-f12601fd3fcc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:12.374540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.374520    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68af80e8-66f0-4d77-903b-f12601fd3fcc-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:12.375873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.375850    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj" pod "68af80e8-66f0-4d77-903b-f12601fd3fcc" (UID: "68af80e8-66f0-4d77-903b-f12601fd3fcc"). InnerVolumeSpecName "kube-api-access-l8vgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:12.470873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.470834    4531 generic.go:356] "Generic (PLEG): container finished" podID="b2a40114-fc1a-4389-a7b3-999bbd0882f0" containerID="2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b" exitCode=0
Nov 01 22:52:12.470873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.470872    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"
Nov 01 22:52:12.471143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.470914    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq" event={"ID":"b2a40114-fc1a-4389-a7b3-999bbd0882f0","Type":"ContainerDied","Data":"2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b"}
Nov 01 22:52:12.471143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.470979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq" event={"ID":"b2a40114-fc1a-4389-a7b3-999bbd0882f0","Type":"ContainerDied","Data":"dcd6d77f57667390ea779dfe1268de4d0c52ad10a3df3e8356e11c2690679f7d"}
Nov 01 22:52:12.471143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.471008    4531 scope.go:122] "RemoveContainer" containerID="2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b"
Nov 01 22:52:12.472607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.472586    4531 generic.go:356] "Generic (PLEG): container finished" podID="68af80e8-66f0-4d77-903b-f12601fd3fcc" containerID="b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda" exitCode=0
Nov 01 22:52:12.472719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.472620    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn" event={"ID":"68af80e8-66f0-4d77-903b-f12601fd3fcc","Type":"ContainerDied","Data":"b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda"}
Nov 01 22:52:12.472719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.472649    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"
Nov 01 22:52:12.472719 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.472658    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn" event={"ID":"68af80e8-66f0-4d77-903b-f12601fd3fcc","Type":"ContainerDied","Data":"37ac54536da255c12b59309adacfc1bb5a40e4d52e1a9febda67c23bfea652f1"}
Nov 01 22:52:12.476102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.475172    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q\" (UniqueName: \"kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q\") pod \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") "
Nov 01 22:52:12.476102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.475289    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume\") pod \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\" (UID: \"b2a40114-fc1a-4389-a7b3-999bbd0882f0\") "
Nov 01 22:52:12.476102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.475711    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume" pod "b2a40114-fc1a-4389-a7b3-999bbd0882f0" (UID: "b2a40114-fc1a-4389-a7b3-999bbd0882f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:52:12.476102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.476048    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2a40114-fc1a-4389-a7b3-999bbd0882f0-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:12.476102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.476072    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8vgj\" (UniqueName: \"kubernetes.io/projected/68af80e8-66f0-4d77-903b-f12601fd3fcc-kube-api-access-l8vgj\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:12.477464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.477438    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q" pod "b2a40114-fc1a-4389-a7b3-999bbd0882f0" (UID: "b2a40114-fc1a-4389-a7b3-999bbd0882f0"). InnerVolumeSpecName "kube-api-access-nds5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:52:12.482521 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.482505    4531 scope.go:122] "RemoveContainer" containerID="2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b"
Nov 01 22:52:12.482903 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:52:12.482874    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b\": container with ID starting with 2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b not found: ID does not exist" containerID="2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b"
Nov 01 22:52:12.482976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.482912    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b"} err="failed to get container status \"2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b\": rpc error: code = NotFound desc = could not find container \"2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b\": container with ID starting with 2447f3748538ce722797922cc00067bbbf44dffb99e85f0baf1a7d3754509a3b not found: ID does not exist"
Nov 01 22:52:12.482976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.482936    4531 scope.go:122] "RemoveContainer" containerID="b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda"
Nov 01 22:52:12.499981 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.499584    4531 scope.go:122] "RemoveContainer" containerID="b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda"
Nov 01 22:52:12.512805 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:52:12.500722    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda\": container with ID starting with b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda not found: ID does not exist" containerID="b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda"
Nov 01 22:52:12.512805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.500779    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda"} err="failed to get container status \"b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda\": rpc error: code = NotFound desc = could not find container \"b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda\": container with ID starting with b4329723e727a728ee24d315bc57975762afaa8e537b631b05594905f95cadda not found: ID does not exist"
Nov 01 22:52:12.535364 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:52:12.535337    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:52:12.577833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.577796    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nds5q\" (UniqueName: \"kubernetes.io/projected/b2a40114-fc1a-4389-a7b3-999bbd0882f0-kube-api-access-nds5q\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:52:12.691593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.691561    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"]
Nov 01 22:52:12.698291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.698263    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-de3d6ade-dfd8988c5-4s8tn"]
Nov 01 22:52:12.820083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.820054    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68af80e8-66f0-4d77-903b-f12601fd3fcc" path="/var/lib/kubelet/pods/68af80e8-66f0-4d77-903b-f12601fd3fcc/volumes"
Nov 01 22:52:12.898856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.898827    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"]
Nov 01 22:52:12.913562 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:12.913532    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-de3d6ade-984c47855-5cdrq"]
Nov 01 22:52:14.331756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:14.331696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:14.819469 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:14.819435    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a40114-fc1a-4389-a7b3-999bbd0882f0" path="/var/lib/kubelet/pods/b2a40114-fc1a-4389-a7b3-999bbd0882f0/volumes"
Nov 01 22:52:19.152941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:19.152882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:23.084604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:23.084564    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:23.703329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:23.703291    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:24.056815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:24.056694    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:24.262460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:24.262428    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:24.479504 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:24.479467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:27.448661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:27.448624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:27.629823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:27.629781    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:27.842187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:27.842108    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:28.044598 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:28.044572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:29.439217 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:29.439181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:32.446426 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:32.446390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:32.649489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:32.649447    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:34.610441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:34.610391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:34.820315 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:34.820276    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:37.379999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:37.379964    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:37.838742 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:37.838694    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:38.428710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:38.428678    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:38.699805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:38.692581    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:39.947150 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:39.947100    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:42.527453 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:52:42.527415    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:52:43.197311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:43.197281    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:43.407200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:43.407139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:45.223892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:45.223857    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:45.418506 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:45.418478    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:48.366178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:48.366137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:50.880803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:50.880312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:51.442771 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:51.442711    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:51.736839 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:51.736789    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:51.884153 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:51.884122    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"]
Nov 01 22:52:51.930022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:51.929993    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-de3d6ade-vwx2m"]
Nov 01 22:52:51.935313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:51.935287    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:52.332189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:52.332159    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:52.819979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:52.819944    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" path="/var/lib/kubelet/pods/4fd49de9-75ff-4d35-b3a4-7c47dc0fb445/volumes"
Nov 01 22:52:57.050949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:57.050915    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"]
Nov 01 22:52:57.062151 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:57.062119    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:57.100693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:57.100660    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5a3e0d9-xkps7"]
Nov 01 22:52:58.261513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:58.261467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:58.482722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:58.482688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:58.685832 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:58.685746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:58.820351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:58.820315    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db21044-330e-4979-b3d3-61bce6111a3c" path="/var/lib/kubelet/pods/4db21044-330e-4979-b3d3-61bce6111a3c/volumes"
Nov 01 22:52:59.466118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:59.466089    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:52:59.676381 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:52:59.676347    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:02.052215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:02.052176    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:06.854089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:06.854038    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:10.265382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:10.265346    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:10.541539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:10.541452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:11.651189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:11.651157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:12.529676 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:53:12.529643    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:53:15.049611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:15.049578    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:16.471250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:16.471219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:21.288857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:21.288774    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:25.988558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:25.988516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:30.654451 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:30.654417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:31.831499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:31.831469    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:35.381029 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:35.380999    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:35.584640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:35.584600    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:35.787544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:35.787510    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:37.744940 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:37.744907    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:38.633449 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:38.633409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:39.107068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:39.105998    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:39.550049 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:39.550019    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:40.264594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:40.264557    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:40.267287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:40.267193    4531 scope.go:122] "RemoveContainer" containerID="066ef917444310e02fa3c3b64f2f928bb5955b9c176a18fbe61620e99fae6683"
Nov 01 22:53:40.642990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:40.642943    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:41.005754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:40.998831    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:42.532837 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:53:42.532800    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:53:44.267237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:44.267187    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:44.610802 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:44.610708    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:46.273704 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:46.273666    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:47.530453 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:47.530417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:47.737965 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:47.737933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:51.752886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:51.752839    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:53.954391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.954358    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"]
Nov 01 22:53:53.955344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955325    4531 cpu_manager.go:414] "containerMap: removing container" podUID="68af80e8-66f0-4d77-903b-f12601fd3fcc" containerName="payload-processing"
Nov 01 22:53:53.955420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955345    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68af80e8-66f0-4d77-903b-f12601fd3fcc" containerName="payload-processing"
Nov 01 22:53:53.955420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955360    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" containerName="revoke-keys"
Nov 01 22:53:53.955420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955366    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" containerName="revoke-keys"
Nov 01 22:53:53.955420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955396    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b2a40114-fc1a-4389-a7b3-999bbd0882f0" containerName="payload-pre-processing"
Nov 01 22:53:53.955420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955402    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a40114-fc1a-4389-a7b3-999bbd0882f0" containerName="payload-pre-processing"
Nov 01 22:53:53.955638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955543    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="68af80e8-66f0-4d77-903b-f12601fd3fcc" containerName="payload-processing"
Nov 01 22:53:53.955638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955554    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd49de9-75ff-4d35-b3a4-7c47dc0fb445" containerName="revoke-keys"
Nov 01 22:53:53.955638 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.955562    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a40114-fc1a-4389-a7b3-999bbd0882f0" containerName="payload-pre-processing"
Nov 01 22:53:53.961249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:53.961227    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.018082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.018058    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"]
Nov 01 22:53:54.057420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.057392    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:53:54.057554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.057420    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:53:54.068871 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.068846    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.069144 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.069116    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.069386 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.069236    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgz8\" (UniqueName: \"kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.170929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.170901    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.171082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.171066    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.171131 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.171104    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgz8\" (UniqueName: \"kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.171779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.171756    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.173376 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.173353    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.225421 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.225364    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgz8\" (UniqueName: \"kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-9dw2m\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.277248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.276997    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:54.662022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:54.661994    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"]
Nov 01 22:53:54.666042 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:53:54.665998    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1025acda_a5fb_4302_90ef_190d0ac8359a.slice/crio-0509555f0c622528f3e654c3360ab1cccc557ffb33c1f9defcb2952fddaaf481 WatchSource:0}: Error finding container 0509555f0c622528f3e654c3360ab1cccc557ffb33c1f9defcb2952fddaaf481: Status 404 returned error can't find the container with id 0509555f0c622528f3e654c3360ab1cccc557ffb33c1f9defcb2952fddaaf481
Nov 01 22:53:55.628968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:55.628935    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:55.689783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:55.687674    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m" event={"ID":"1025acda-a5fb-4302-90ef-190d0ac8359a","Type":"ContainerStarted","Data":"8b05f3d53f5e5b4aeb44e20a6a11e7e8f5e3e82085faa8e72df4f71791ccb527"}
Nov 01 22:53:55.689783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:55.687793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m" event={"ID":"1025acda-a5fb-4302-90ef-190d0ac8359a","Type":"ContainerStarted","Data":"0509555f0c622528f3e654c3360ab1cccc557ffb33c1f9defcb2952fddaaf481"}
Nov 01 22:53:55.796375 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:55.796325    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m" podStartSLOduration=2.7963119880000002 podStartE2EDuration="2.796311988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:53:55.795019297 +0000 UTC m=+1817.112669301" watchObservedRunningTime="2026-11-01 22:53:55.796311988 +0000 UTC m=+1817.113961989"
Nov 01 22:53:56.050660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:56.050633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:56.651369 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:56.651340    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:56.823747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:56.817451    4531 generic.go:356] "Generic (PLEG): container finished" podID="1025acda-a5fb-4302-90ef-190d0ac8359a" containerID="8b05f3d53f5e5b4aeb44e20a6a11e7e8f5e3e82085faa8e72df4f71791ccb527" exitCode=0
Nov 01 22:53:57.091750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:57.075337    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:57.091750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:57.090623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m" event={"ID":"1025acda-a5fb-4302-90ef-190d0ac8359a","Type":"ContainerDied","Data":"8b05f3d53f5e5b4aeb44e20a6a11e7e8f5e3e82085faa8e72df4f71791ccb527"}
Nov 01 22:53:57.261133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:57.261099    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:57.452201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:57.452160    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:53:58.034675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.034652    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:58.078861 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.078824    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca\") pod \"1025acda-a5fb-4302-90ef-190d0ac8359a\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") "
Nov 01 22:53:58.079023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.078874    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8\" (UniqueName: \"kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8\") pod \"1025acda-a5fb-4302-90ef-190d0ac8359a\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") "
Nov 01 22:53:58.079023 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.078923    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca\") pod \"1025acda-a5fb-4302-90ef-190d0ac8359a\" (UID: \"1025acda-a5fb-4302-90ef-190d0ac8359a\") "
Nov 01 22:53:58.079391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.079352    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca" pod "1025acda-a5fb-4302-90ef-190d0ac8359a" (UID: "1025acda-a5fb-4302-90ef-190d0ac8359a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:53:58.079961 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.079937    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:58.081365 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.081318    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8" pod "1025acda-a5fb-4302-90ef-190d0ac8359a" (UID: "1025acda-a5fb-4302-90ef-190d0ac8359a"). InnerVolumeSpecName "kube-api-access-rcgz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:53:58.081696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.081669    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca" pod "1025acda-a5fb-4302-90ef-190d0ac8359a" (UID: "1025acda-a5fb-4302-90ef-190d0ac8359a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:53:58.182013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.181970    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1025acda-a5fb-4302-90ef-190d0ac8359a-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:58.182013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.182009    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgz8\" (UniqueName: \"kubernetes.io/projected/1025acda-a5fb-4302-90ef-190d0ac8359a-kube-api-access-rcgz8\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:53:58.848347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.848307    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m" event={"ID":"1025acda-a5fb-4302-90ef-190d0ac8359a","Type":"ContainerDied","Data":"0509555f0c622528f3e654c3360ab1cccc557ffb33c1f9defcb2952fddaaf481"}
Nov 01 22:53:58.848532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.848361    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0509555f0c622528f3e654c3360ab1cccc557ffb33c1f9defcb2952fddaaf481"
Nov 01 22:53:58.848532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:58.848411    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"
Nov 01 22:53:59.695025 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.694996    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"]
Nov 01 22:53:59.695767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.695745    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1025acda-a5fb-4302-90ef-190d0ac8359a" containerName="revoke-keys"
Nov 01 22:53:59.695767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.695762    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1025acda-a5fb-4302-90ef-190d0ac8359a" containerName="revoke-keys"
Nov 01 22:53:59.695944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.695887    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1025acda-a5fb-4302-90ef-190d0ac8359a" containerName="revoke-keys"
Nov 01 22:53:59.699883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.699863    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.828455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.828419    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9r8s\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.828939 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.828918    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.829254 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.829238    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.829442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.829427    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.829708 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.829676    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.829897 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.829882    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.833209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.833159    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.833423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.833397    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.833549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.833535    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.872644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.872613    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8bac2f-openshift-default-dockercfg-dvrdf\""
Nov 01 22:53:59.911441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.911095    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"]
Nov 01 22:53:59.935041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.934830    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.934913    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.934974    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.935021    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.935057    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.935112    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9r8s\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.935187    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.935279    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.935402 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.935368    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.937047 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.936874    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.945467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.937628    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.945467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.938556    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.945467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.940639    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.945467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.940947    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.945467 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.943664    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.947587 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.947563    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.991343 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.991304    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:53:59.992958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:53:59.992929    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9r8s\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:54:00.015694 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.015656    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:54:00.452416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.452391    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"]
Nov 01 22:54:00.455265 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:54:00.455239    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72da0505_3e3f_4764_a695_faa46ae5a8d3.slice/crio-79537ed3ea844aed38c882d548ad59a11c546baf6e4f05fed6987e2019e99e03 WatchSource:0}: Error finding container 79537ed3ea844aed38c882d548ad59a11c546baf6e4f05fed6987e2019e99e03: Status 404 returned error can't find the container with id 79537ed3ea844aed38c882d548ad59a11c546baf6e4f05fed6987e2019e99e03
Nov 01 22:54:00.457872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.457836    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:54:00.457977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.457918    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:54:00.457977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.457965    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:54:00.935085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.934992    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" event={"ID":"72da0505-3e3f-4764-a695-faa46ae5a8d3","Type":"ContainerStarted","Data":"427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c"}
Nov 01 22:54:00.935085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:00.935057    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" event={"ID":"72da0505-3e3f-4764-a695-faa46ae5a8d3","Type":"ContainerStarted","Data":"79537ed3ea844aed38c882d548ad59a11c546baf6e4f05fed6987e2019e99e03"}
Nov 01 22:54:01.016393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:01.016362    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:54:01.017646 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:01.017623    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:01.017787 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:01.017670    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:01.046389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:01.046341    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podStartSLOduration=2.046326741 podStartE2EDuration="2.046326741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:01.045400512 +0000 UTC m=+1822.363050514" watchObservedRunningTime="2026-11-01 22:54:01.046326741 +0000 UTC m=+1822.363976742"
Nov 01 22:54:01.479237 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:01.479195    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:01.694474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:01.694269    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:02.016541 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:02.016501    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:02.017020 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:02.016569    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:02.565759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:02.565355    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:03.016473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:03.016309    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:03.016473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:03.016370    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:04.016209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:04.016178    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:04.016391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:04.016237    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:04.022992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:04.022958    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"]
Nov 01 22:54:04.023428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:04.023166    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" containerID="cri-o://0b5b9035f4d435910e03f08f87f43944443e3505a8338144b5018fe4afae0f71" gracePeriod=30
Nov 01 22:54:05.016971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:05.016930    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:05.017145 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:05.016989    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:05.893757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:05.893077    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:06.030753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:06.016145    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:06.030753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:06.016205    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:06.152630 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:06.152476    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:07.016635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:07.016592    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body=
Nov 01 22:54:07.017201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:07.016678    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused"
Nov 01 22:54:08.021653 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:08.021620    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:54:08.022128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:08.022115    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:54:08.022908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:08.022885    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 22:54:08.172649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:08.172576    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:09.012110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.012077    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:54:09.012298 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.012147    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:54:09.283311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.283229    4531 generic.go:356] "Generic (PLEG): container finished" podID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerID="0b5b9035f4d435910e03f08f87f43944443e3505a8338144b5018fe4afae0f71" exitCode=0
Nov 01 22:54:09.283898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.283831    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" event={"ID":"85b0a533-ebfb-4a51-8898-60d0c00e4748","Type":"ContainerDied","Data":"0b5b9035f4d435910e03f08f87f43944443e3505a8338144b5018fe4afae0f71"}
Nov 01 22:54:09.344075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.344046    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510326    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510419    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510499    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510531    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510575    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510621    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510676    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510816    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.510888    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy\") pod \"85b0a533-ebfb-4a51-8898-60d0c00e4748\" (UID: \"85b0a533-ebfb-4a51-8898-60d0c00e4748\") "
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.512640    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.512895    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.514082 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.513119    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.515109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.514957    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:09.515293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.515259    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.517634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.517610    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:09.519596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.519574    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:09.519741 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.519659    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:09.519846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.519788    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm" pod "85b0a533-ebfb-4a51-8898-60d0c00e4748" (UID: "85b0a533-ebfb-4a51-8898-60d0c00e4748"). InnerVolumeSpecName "kube-api-access-rq4rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:09.613289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613204    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613234    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613248    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/85b0a533-ebfb-4a51-8898-60d0c00e4748-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613261    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613274    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613286    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613303    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/85b0a533-ebfb-4a51-8898-60d0c00e4748-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613315    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq4rm\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-kube-api-access-rq4rm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:09.613620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:09.613327    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/85b0a533-ebfb-4a51-8898-60d0c00e4748-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:10.262864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.262830    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"]
Nov 01 22:54:10.293068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.293038    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"
Nov 01 22:54:10.293599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.293068    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f" event={"ID":"85b0a533-ebfb-4a51-8898-60d0c00e4748","Type":"ContainerDied","Data":"a2f5d0f40a8c56004009ea5cfab96ef87f8ffe925c31e64768ed1f399a801f64"}
Nov 01 22:54:10.293599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.293124    4531 scope.go:122] "RemoveContainer" containerID="0b5b9035f4d435910e03f08f87f43944443e3505a8338144b5018fe4afae0f71"
Nov 01 22:54:10.501685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.501647    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"]
Nov 01 22:54:10.545599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.545530    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-zr46f"]
Nov 01 22:54:10.569668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.569635    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:10.773071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.773035    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:10.827102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:10.827013    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" path="/var/lib/kubelet/pods/85b0a533-ebfb-4a51-8898-60d0c00e4748/volumes"
Nov 01 22:54:12.534897 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:54:12.534863    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:54:13.310994 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:13.310958    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:13.498155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:13.498116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:14.666502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:14.666467    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:14.955094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:14.955056    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:15.264210 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:15.264122    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerName="payload-pre-processing" containerID="cri-o://234d0bf7060d48dd64215bd48635849fbe0c73eb3317d116cc4d79c65aab74ca" gracePeriod=25
Nov 01 22:54:16.038463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:16.038430    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"]
Nov 01 22:54:16.038910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:16.038638    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" containerID="cri-o://45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c" gracePeriod=30
Nov 01 22:54:17.880571 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.880536    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"]
Nov 01 22:54:17.881615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.881592    4531 cpu_manager.go:414] "containerMap: removing container" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy"
Nov 01 22:54:17.881615 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.881613    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy"
Nov 01 22:54:17.881880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.881808    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b0a533-ebfb-4a51-8898-60d0c00e4748" containerName="istio-proxy"
Nov 01 22:54:17.887674 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.887655    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:17.940754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.940694    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"]
Nov 01 22:54:17.952663 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.952630    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:17.952862 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.952842    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6tmv\" (UniqueName: \"kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:17.952934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.952919    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:17.979907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.979882    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:54:17.980071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:17.979950    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:54:18.053796 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.053766    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6tmv\" (UniqueName: \"kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.053908 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.053871    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.053971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.053947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.054570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.054548    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.056147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.056126    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.106633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.106608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6tmv\" (UniqueName: \"kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv\") pod \"maas-api-revoke-sub-4d59ac06-w9nqc\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.203986 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.203965    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:18.581100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.581048    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"]
Nov 01 22:54:18.584472 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:54:18.584444    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd13d9553_ea54_4cc6_a5e5_8f48d80a3c6c.slice/crio-eb11bb801a93af3a4e37979a537a0168704eb6ada949524c22993720dd229bc4 WatchSource:0}: Error finding container eb11bb801a93af3a4e37979a537a0168704eb6ada949524c22993720dd229bc4: Status 404 returned error can't find the container with id eb11bb801a93af3a4e37979a537a0168704eb6ada949524c22993720dd229bc4
Nov 01 22:54:18.632448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:18.632420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:19.371555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:19.371517    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc" event={"ID":"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c","Type":"ContainerStarted","Data":"bd76a7ba249f168542ddbf9df0c427204daef4a2057f513ab8627cd055703f1c"}
Nov 01 22:54:19.372085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:19.371590    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc" event={"ID":"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c","Type":"ContainerStarted","Data":"eb11bb801a93af3a4e37979a537a0168704eb6ada949524c22993720dd229bc4"}
Nov 01 22:54:19.482575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:19.482488    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc" podStartSLOduration=2.482471339 podStartE2EDuration="2.482471339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:19.480176781 +0000 UTC m=+1840.797826783" watchObservedRunningTime="2026-11-01 22:54:19.482471339 +0000 UTC m=+1840.800121342"
Nov 01 22:54:20.293110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:20.293072    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:54:20.293285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:20.293138    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:54:20.381781 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:20.381743    4531 generic.go:356] "Generic (PLEG): container finished" podID="d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" containerID="bd76a7ba249f168542ddbf9df0c427204daef4a2057f513ab8627cd055703f1c" exitCode=0
Nov 01 22:54:20.382189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:20.381819    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc" event={"ID":"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c","Type":"ContainerDied","Data":"bd76a7ba249f168542ddbf9df0c427204daef4a2057f513ab8627cd055703f1c"}
Nov 01 22:54:21.130518 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.130482    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:21.323571 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.323516    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:54:21.324268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.324237    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:21.391169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.391141    4531 generic.go:356] "Generic (PLEG): container finished" podID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerID="45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c" exitCode=0
Nov 01 22:54:21.391559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.391185    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"
Nov 01 22:54:21.391559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.391226    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" event={"ID":"1dbb1c78-2cc1-474a-9734-420b9be02178","Type":"ContainerDied","Data":"45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c"}
Nov 01 22:54:21.391559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.391292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl" event={"ID":"1dbb1c78-2cc1-474a-9734-420b9be02178","Type":"ContainerDied","Data":"b2f7512ba43f85baf8d49dbde6db34ec85a5d7c9adb4a9a118fb150aba7191d2"}
Nov 01 22:54:21.391559 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.391325    4531 scope.go:122] "RemoveContainer" containerID="45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c"
Nov 01 22:54:21.399989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.399966    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400006    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400092    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400173 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400155    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400176    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400251    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400279 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400262    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.400423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400303    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400345    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.400423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400388    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert\") pod \"1dbb1c78-2cc1-474a-9734-420b9be02178\" (UID: \"1dbb1c78-2cc1-474a-9734-420b9be02178\") "
Nov 01 22:54:21.401008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400767    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.401008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.400790    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.401174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.401012    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.401255 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.401232    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.401319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.401261    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.401319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.401278    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.401319 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.401292    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.401464 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.401371    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:21.403257 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.403212    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.404152 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.404127    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:54:21.404251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.404223    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "kube-api-access-b4nqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.404809 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.404784    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo" pod "1dbb1c78-2cc1-474a-9734-420b9be02178" (UID: "1dbb1c78-2cc1-474a-9734-420b9be02178"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:54:21.404942 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.404922    4531 scope.go:122] "RemoveContainer" containerID="45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c"
Nov 01 22:54:21.405509 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:54:21.405487    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c\": container with ID starting with 45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c not found: ID does not exist" containerID="45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c"
Nov 01 22:54:21.405583 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.405518    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c"} err="failed to get container status \"45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c\": rpc error: code = NotFound desc = could not find container \"45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c\": container with ID starting with 45e7b144f84749993061c0df9823b496fdbfc85dff0a1e605271dc4f22b85c9c not found: ID does not exist"
Nov 01 22:54:21.507604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.507547    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.507604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.507585    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1dbb1c78-2cc1-474a-9734-420b9be02178-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.507604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.507600    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4nqc\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-kube-api-access-b4nqc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.507929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.507615    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.507929 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.507630    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1dbb1c78-2cc1-474a-9734-420b9be02178-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.562918 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.562888    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:21.710089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.710051    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca\") pod \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") "
Nov 01 22:54:21.710247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.710142    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv\" (UniqueName: \"kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv\") pod \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") "
Nov 01 22:54:21.710343 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.710321    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca\") pod \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\" (UID: \"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c\") "
Nov 01 22:54:21.710723 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.710690    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca" pod "d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" (UID: "d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:21.711520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.711492    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.712265 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.712236    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca" pod "d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" (UID: "d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:54:21.712364 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.712310    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv" pod "d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" (UID: "d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c"). InnerVolumeSpecName "kube-api-access-n6tmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:21.813176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.813091    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.813176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.813127    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6tmv\" (UniqueName: \"kubernetes.io/projected/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c-kube-api-access-n6tmv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:21.828230 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.827961    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:54:21.831060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.831034    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"]
Nov 01 22:54:21.842675 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:21.842647    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-zv6tl"]
Nov 01 22:54:22.405985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:22.405913    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"
Nov 01 22:54:22.406363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:22.405906    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc" event={"ID":"d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c","Type":"ContainerDied","Data":"eb11bb801a93af3a4e37979a537a0168704eb6ada949524c22993720dd229bc4"}
Nov 01 22:54:22.406363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:22.406051    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb11bb801a93af3a4e37979a537a0168704eb6ada949524c22993720dd229bc4"
Nov 01 22:54:22.818971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:22.818928    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" path="/var/lib/kubelet/pods/1dbb1c78-2cc1-474a-9734-420b9be02178/volumes"
Nov 01 22:54:23.421952 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:23.421915    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:28.153029 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.152925    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:28.251069 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251036    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"]
Nov 01 22:54:28.251761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251747    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy"
Nov 01 22:54:28.251844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251762    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy"
Nov 01 22:54:28.251844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251814    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" containerName="revoke-keys"
Nov 01 22:54:28.251844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251819    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" containerName="revoke-keys"
Nov 01 22:54:28.251951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251919    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" containerName="revoke-keys"
Nov 01 22:54:28.251951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.251931    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbb1c78-2cc1-474a-9734-420b9be02178" containerName="istio-proxy"
Nov 01 22:54:28.256779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.256759    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.310292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.310267    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"]
Nov 01 22:54:28.350312 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.350284    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:54:28.350458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.350386    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:54:28.403508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.403434    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.403508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.403477    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.403766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.403749    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9xsk\" (UniqueName: \"kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.505565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.505536    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9xsk\" (UniqueName: \"kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.505756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.505702    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.505756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.505747    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.506403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.506358    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.508420 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.508391    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.558926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.558897    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9xsk\" (UniqueName: \"kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk\") pod \"maas-api-revoke-sub-244c5c8d-j4dqz\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.572762 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.572739    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:28.980370 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:28.980340    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"]
Nov 01 22:54:28.981462 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:54:28.981432    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f57916e_eed6_43fd_b5d7_90d347603064.slice/crio-c7a3d556022350f25a9ca96776498de084ff477a5d1609684a337a37d012cc34 WatchSource:0}: Error finding container c7a3d556022350f25a9ca96776498de084ff477a5d1609684a337a37d012cc34: Status 404 returned error can't find the container with id c7a3d556022350f25a9ca96776498de084ff477a5d1609684a337a37d012cc34
Nov 01 22:54:29.466336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:29.466301    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz" event={"ID":"6f57916e-eed6-43fd-b5d7-90d347603064","Type":"ContainerStarted","Data":"81d9d52075b4fcbfc69a3476b9ab728240b83fbaf0f1b3c3406e8b654b8bf8be"}
Nov 01 22:54:29.466826 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:29.466355    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz" event={"ID":"6f57916e-eed6-43fd-b5d7-90d347603064","Type":"ContainerStarted","Data":"c7a3d556022350f25a9ca96776498de084ff477a5d1609684a337a37d012cc34"}
Nov 01 22:54:29.579275 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:29.579194    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz" podStartSLOduration=1.579180983 podStartE2EDuration="1.579180983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:29.573792356 +0000 UTC m=+1850.891442358" watchObservedRunningTime="2026-11-01 22:54:29.579180983 +0000 UTC m=+1850.896830984"
Nov 01 22:54:30.476292 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:30.476257    4531 generic.go:356] "Generic (PLEG): container finished" podID="6f57916e-eed6-43fd-b5d7-90d347603064" containerID="81d9d52075b4fcbfc69a3476b9ab728240b83fbaf0f1b3c3406e8b654b8bf8be" exitCode=0
Nov 01 22:54:30.476711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:30.476343    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz" event={"ID":"6f57916e-eed6-43fd-b5d7-90d347603064","Type":"ContainerDied","Data":"81d9d52075b4fcbfc69a3476b9ab728240b83fbaf0f1b3c3406e8b654b8bf8be"}
Nov 01 22:54:31.669806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.669782    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:31.749146 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.749086    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca\") pod \"6f57916e-eed6-43fd-b5d7-90d347603064\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") "
Nov 01 22:54:31.749295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.749160    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk\" (UniqueName: \"kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk\") pod \"6f57916e-eed6-43fd-b5d7-90d347603064\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") "
Nov 01 22:54:31.749395 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.749364    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca\") pod \"6f57916e-eed6-43fd-b5d7-90d347603064\" (UID: \"6f57916e-eed6-43fd-b5d7-90d347603064\") "
Nov 01 22:54:31.749495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.749475    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca" pod "6f57916e-eed6-43fd-b5d7-90d347603064" (UID: "6f57916e-eed6-43fd-b5d7-90d347603064"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:31.749950 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.749929    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:31.751233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.751205    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk" pod "6f57916e-eed6-43fd-b5d7-90d347603064" (UID: "6f57916e-eed6-43fd-b5d7-90d347603064"). InnerVolumeSpecName "kube-api-access-z9xsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:31.751317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.751252    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca" pod "6f57916e-eed6-43fd-b5d7-90d347603064" (UID: "6f57916e-eed6-43fd-b5d7-90d347603064"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:54:31.829625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.829589    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 22:54:31.851624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.851598    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9xsk\" (UniqueName: \"kubernetes.io/projected/6f57916e-eed6-43fd-b5d7-90d347603064-kube-api-access-z9xsk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:31.851624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:31.851622    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f57916e-eed6-43fd-b5d7-90d347603064-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:32.497155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:32.497121    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"
Nov 01 22:54:32.497334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:32.497161    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz" event={"ID":"6f57916e-eed6-43fd-b5d7-90d347603064","Type":"ContainerDied","Data":"c7a3d556022350f25a9ca96776498de084ff477a5d1609684a337a37d012cc34"}
Nov 01 22:54:32.497334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:32.497217    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7a3d556022350f25a9ca96776498de084ff477a5d1609684a337a37d012cc34"
Nov 01 22:54:32.984189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:32.984150    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:35.554760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:35.554658    4531 generic.go:356] "Generic (PLEG): container finished" podID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerID="234d0bf7060d48dd64215bd48635849fbe0c73eb3317d116cc4d79c65aab74ca" exitCode=0
Nov 01 22:54:35.555200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:35.554979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" event={"ID":"01933a1d-f834-46d2-81ca-a6ae157a4800","Type":"ContainerDied","Data":"234d0bf7060d48dd64215bd48635849fbe0c73eb3317d116cc4d79c65aab74ca"}
Nov 01 22:54:36.152911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.152880    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:54:36.279783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.279444    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume\") pod \"01933a1d-f834-46d2-81ca-a6ae157a4800\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") "
Nov 01 22:54:36.279783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.279774    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6\" (UniqueName: \"kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6\") pod \"01933a1d-f834-46d2-81ca-a6ae157a4800\" (UID: \"01933a1d-f834-46d2-81ca-a6ae157a4800\") "
Nov 01 22:54:36.281118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.281090    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume" pod "01933a1d-f834-46d2-81ca-a6ae157a4800" (UID: "01933a1d-f834-46d2-81ca-a6ae157a4800"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:54:36.297983 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.297919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6" pod "01933a1d-f834-46d2-81ca-a6ae157a4800" (UID: "01933a1d-f834-46d2-81ca-a6ae157a4800"). InnerVolumeSpecName "kube-api-access-45vh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:54:36.386122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.386052    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45vh6\" (UniqueName: \"kubernetes.io/projected/01933a1d-f834-46d2-81ca-a6ae157a4800-kube-api-access-45vh6\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:36.386122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.386094    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01933a1d-f834-46d2-81ca-a6ae157a4800-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:54:36.583336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.583225    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc" event={"ID":"01933a1d-f834-46d2-81ca-a6ae157a4800","Type":"ContainerDied","Data":"a7f7e4b8d15f21e70ffd8e96f95c037e3d34e69f6340f8cbb795b1c45bac2d3d"}
Nov 01 22:54:36.583932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.583895    4531 scope.go:122] "RemoveContainer" containerID="234d0bf7060d48dd64215bd48635849fbe0c73eb3317d116cc4d79c65aab74ca"
Nov 01 22:54:36.584159 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.583940    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"
Nov 01 22:54:36.743468 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.743432    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"]
Nov 01 22:54:36.790668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.790632    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-945efd02-67c88444dc-t9kmc"]
Nov 01 22:54:36.823091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:36.823054    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" path="/var/lib/kubelet/pods/01933a1d-f834-46d2-81ca-a6ae157a4800/volumes"
Nov 01 22:54:38.079338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:38.079296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:38.284772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:38.284719    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:39.035489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:39.035457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:40.529875 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:40.529844    4531 scope.go:122] "RemoveContainer" containerID="bbc35ebf975128d63b17c7879ad0f5741e25d5e4753fe134eec752dc9948a9c9"
Nov 01 22:54:40.541019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:40.541001    4531 scope.go:122] "RemoveContainer" containerID="e1af48c69fe5c2583856afd2774386897fdb015c981a063947d8e3bf1aa40d93"
Nov 01 22:54:40.551095 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:40.551075    4531 scope.go:122] "RemoveContainer" containerID="405f6e1e2a90ee63f59cfe42630a74a668f9106eac8549643c98e3b9f9137f94"
Nov 01 22:54:42.527083 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:54:42.527050    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:54:42.750104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:42.750063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:45.098342 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.098307    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"]
Nov 01 22:54:45.099068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.099051    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6f57916e-eed6-43fd-b5d7-90d347603064" containerName="revoke-keys"
Nov 01 22:54:45.099068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.099068    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f57916e-eed6-43fd-b5d7-90d347603064" containerName="revoke-keys"
Nov 01 22:54:45.099203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.099088    4531 cpu_manager.go:414] "containerMap: removing container" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerName="payload-pre-processing"
Nov 01 22:54:45.099203 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.099094    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerName="payload-pre-processing"
Nov 01 22:54:45.099268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.099205    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f57916e-eed6-43fd-b5d7-90d347603064" containerName="revoke-keys"
Nov 01 22:54:45.099268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.099216    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="01933a1d-f834-46d2-81ca-a6ae157a4800" containerName="payload-pre-processing"
Nov 01 22:54:45.105233 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.105208    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224305 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrrc\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224384    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224446    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224623 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224553    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224623 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224576    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224661    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.224931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224912    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.225061 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.224946    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.304329 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.304294    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"]
Nov 01 22:54:45.326528 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.326497    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.326528 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.326531    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.326764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.326606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.326764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.326625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.326764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.326656    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327144    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327268 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327222    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327333 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327308    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327350    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327483 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327468    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrrc\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327552 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327531    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327468    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327720 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327680    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.327955 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.327933    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.329235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.329208    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.329778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.329718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.380596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.380525    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.380774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.380747    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrrc\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.419739 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.419706    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:45.450200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.450167    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:45.862409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.862380    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"]
Nov 01 22:54:45.865698 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:54:45.865673    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf010fe17_8581_43df_9211_5b59979b9fe7.slice/crio-52d9aabdc8aa5dde24ff76cf56b69598d780584a7f8d0b81ab944e3c39cd86ad WatchSource:0}: Error finding container 52d9aabdc8aa5dde24ff76cf56b69598d780584a7f8d0b81ab944e3c39cd86ad: Status 404 returned error can't find the container with id 52d9aabdc8aa5dde24ff76cf56b69598d780584a7f8d0b81ab944e3c39cd86ad
Nov 01 22:54:45.868302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.868271    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:54:45.868385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.868333    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:54:45.868385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:45.868363    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:54:46.693764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:46.693712    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" event={"ID":"f010fe17-8581-43df-9211-5b59979b9fe7","Type":"ContainerStarted","Data":"6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4"}
Nov 01 22:54:46.694181 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:46.693791    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" event={"ID":"f010fe17-8581-43df-9211-5b59979b9fe7","Type":"ContainerStarted","Data":"52d9aabdc8aa5dde24ff76cf56b69598d780584a7f8d0b81ab944e3c39cd86ad"}
Nov 01 22:54:46.900553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:46.900495    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podStartSLOduration=1.9004813409999999 podStartE2EDuration="1.900481341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:54:46.853118388 +0000 UTC m=+1868.170768389" watchObservedRunningTime="2026-11-01 22:54:46.900481341 +0000 UTC m=+1868.218131342"
Nov 01 22:54:47.420594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:47.420562    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:47.422560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:47.422533    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:54:47.422694 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:47.422578    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 22:54:47.577752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:47.569360    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:48.420508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:48.420473    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:54:48.420924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:48.420532    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 22:54:49.420626 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:49.420591    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:54:49.421124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:49.420650    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 22:54:50.420065 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:50.420030    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:54:50.420261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:50.420097    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 22:54:51.420497 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:51.420452    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body=
Nov 01 22:54:51.420939 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:51.420524    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused"
Nov 01 22:54:52.424621 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:52.424583    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:52.425101 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:52.425081    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:52.425963 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:52.425944    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:54:52.608658 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:52.608620    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:55.871452 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:55.871415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:56.532898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:56.532862    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:57.484294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:57.484251    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:57.755970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:57.755868    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:58.165403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:58.165314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:58.366071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:58.366032    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:54:58.581031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:54:58.580997    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:00.197785 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.197750    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"]
Nov 01 22:55:00.203668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.203645    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227177 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227148    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227390 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227366    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227423    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227510    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wvr\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227583    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227623    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227768    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227840    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.227915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.227907    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.338749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.338696    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"]
Nov 01 22:55:00.339857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.339824    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.343932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.343902    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.344126 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.344109    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.344236 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.344216    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wvr\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.344348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.344335    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.344473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.344458    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.344661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.344646    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.345208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.345188    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.345387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.345371    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.346004 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.345981    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.346382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.346363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.346765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.346715    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.347487 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.347466    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.351985 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.351960    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.356770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.356705    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.358351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.358328    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.408355 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.408313    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.409739 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.409692    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wvr\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr\") pod \"e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") " pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.521475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.521414    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:00.958610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.958410    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"]
Nov 01 22:55:00.961369 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:55:00.961342    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495700ae_9b3d_47c4_ad34_3d34aad3a218.slice/crio-44ca55349790eb613f379847a1dc2a2be70c27dccc5970ad22159018db536305 WatchSource:0}: Error finding container 44ca55349790eb613f379847a1dc2a2be70c27dccc5970ad22159018db536305: Status 404 returned error can't find the container with id 44ca55349790eb613f379847a1dc2a2be70c27dccc5970ad22159018db536305
Nov 01 22:55:00.964388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.964349    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:55:00.964474 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.964451    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:55:00.964526 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:00.964492    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:55:01.843338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:01.843299    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" event={"ID":"495700ae-9b3d-47c4-ad34-3d34aad3a218","Type":"ContainerStarted","Data":"322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696"}
Nov 01 22:55:01.843750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:01.843360    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" event={"ID":"495700ae-9b3d-47c4-ad34-3d34aad3a218","Type":"ContainerStarted","Data":"44ca55349790eb613f379847a1dc2a2be70c27dccc5970ad22159018db536305"}
Nov 01 22:55:02.054102 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.054050    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podStartSLOduration=2.054034903 podStartE2EDuration="2.054034903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:55:02.003044292 +0000 UTC m=+1883.320694293" watchObservedRunningTime="2026-11-01 22:55:02.054034903 +0000 UTC m=+1883.371684975"
Nov 01 22:55:02.077761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.077377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:02.312575 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.312540    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:02.522314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.522283    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:02.523945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.523914    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 22:55:02.524083 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.523972    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 22:55:02.526350 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:02.526328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:03.521805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:03.521763    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 22:55:03.522227 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:03.521833    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 22:55:03.942593 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:03.942502    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:04.169271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:04.167223    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:04.505750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:04.505283    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:04.532902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:04.524949    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 22:55:04.532902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:04.525009    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 22:55:05.525747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:05.525634    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 22:55:05.525747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:05.525710    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 22:55:06.522620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:06.522583    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 22:55:06.523053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:06.522644    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 22:55:07.017019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:07.016988    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:07.272052 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:07.271927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:07.530349 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:07.530058    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused" start-of-body=
Nov 01 22:55:07.531130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:07.531098    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.167:15021/healthz/ready\": dial tcp 10.133.0.167:15021: connect: connection refused"
Nov 01 22:55:08.224648 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:08.224609    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:08.423907 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:08.423874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:08.526775 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:08.526690    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:08.527002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:08.526983    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:08.527760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:08.527701    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:55:09.410285 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:09.409181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:12.530465 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:55:12.530426    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:55:14.426066 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:14.426017    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:14.608974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:14.608939    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:14.807892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:14.807857    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:20.514744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:20.511628    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:20.744298 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:20.744265    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:21.638858 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:21.638812    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:22.088558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:22.088524    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:22.925393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:22.924363    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:23.140534 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:23.140492    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:26.049956 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:26.049922    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:27.977056 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:27.977010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:28.462712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:28.462672    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:31.506861 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:31.506822    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:34.694530 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:34.694487    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:34.905155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:34.905121    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:36.949291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:36.949233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:38.504118 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:38.504068    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:41.972804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:41.972768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:42.528960 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:55:42.528924    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:55:46.227520 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:46.227484    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:47.850014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:47.849975    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:48.134062 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:48.133955    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:48.331989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:48.331957    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"]
Nov 01 22:55:48.376844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:48.376814    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-9dw2m"]
Nov 01 22:55:48.418022 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:48.417947    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:48.818550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:48.818518    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1025acda-a5fb-4302-90ef-190d0ac8359a" path="/var/lib/kubelet/pods/1025acda-a5fb-4302-90ef-190d0ac8359a/volumes"
Nov 01 22:55:52.105387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:52.105299    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:53.175093 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:53.175049    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:55.612440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:55.612405    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:55.877798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:55.877696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:55:58.371014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:55:58.370976    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:00.311591 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:00.311554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:00.523192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:00.523157    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:03.491769 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:03.491713    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:08.603403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:08.603359    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:09.701021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.700985    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"]
Nov 01 22:56:09.706753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.706717    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.755423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.755395    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:56:09.758039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.758011    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"]
Nov 01 22:56:09.797968 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.797944    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:56:09.820563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.820530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.820695 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.820567    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqr59\" (UniqueName: \"kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.820695 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.820668    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.922122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.922091    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.922122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.922123    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqr59\" (UniqueName: \"kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.922336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.922193    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.923060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.923027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.924943 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.924926    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:09.976414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:09.976358    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqr59\" (UniqueName: \"kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59\") pod \"maas-api-revoke-sub-05f6bd6c-k9wvm\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:10.021356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:10.021330    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:10.418356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:10.418332    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"]
Nov 01 22:56:10.421514 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:56:10.421484    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd65dd531_32d4_4276_960d_471d56d531ee.slice/crio-48db25602d036896b78674ff9bb09b1de236278e3c4369c92fb177fcc8ca528d WatchSource:0}: Error finding container 48db25602d036896b78674ff9bb09b1de236278e3c4369c92fb177fcc8ca528d: Status 404 returned error can't find the container with id 48db25602d036896b78674ff9bb09b1de236278e3c4369c92fb177fcc8ca528d
Nov 01 22:56:10.759192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:10.759164    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm" event={"ID":"d65dd531-32d4-4276-960d-471d56d531ee","Type":"ContainerStarted","Data":"19605a59df22dba530b321b414a0d4343c1f4703223cb907470f7e3e9fff1c11"}
Nov 01 22:56:10.759579 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:10.759215    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm" event={"ID":"d65dd531-32d4-4276-960d-471d56d531ee","Type":"ContainerStarted","Data":"48db25602d036896b78674ff9bb09b1de236278e3c4369c92fb177fcc8ca528d"}
Nov 01 22:56:10.873565 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:10.873521    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm" podStartSLOduration=1.873507839 podStartE2EDuration="1.873507839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:10.870363419 +0000 UTC m=+1952.188013423" watchObservedRunningTime="2026-11-01 22:56:10.873507839 +0000 UTC m=+1952.191157840"
Nov 01 22:56:11.771076 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:11.771044    4531 generic.go:356] "Generic (PLEG): container finished" podID="d65dd531-32d4-4276-960d-471d56d531ee" containerID="19605a59df22dba530b321b414a0d4343c1f4703223cb907470f7e3e9fff1c11" exitCode=0
Nov 01 22:56:11.771490 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:11.771146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm" event={"ID":"d65dd531-32d4-4276-960d-471d56d531ee","Type":"ContainerDied","Data":"19605a59df22dba530b321b414a0d4343c1f4703223cb907470f7e3e9fff1c11"}
Nov 01 22:56:12.543333 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:12.543292    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:56:12.956481 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:12.956458    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:13.064346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.064308    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca\") pod \"d65dd531-32d4-4276-960d-471d56d531ee\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") "
Nov 01 22:56:13.064513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.064378    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca\") pod \"d65dd531-32d4-4276-960d-471d56d531ee\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") "
Nov 01 22:56:13.064608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.064522    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59\" (UniqueName: \"kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59\") pod \"d65dd531-32d4-4276-960d-471d56d531ee\" (UID: \"d65dd531-32d4-4276-960d-471d56d531ee\") "
Nov 01 22:56:13.064857 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.064824    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca" pod "d65dd531-32d4-4276-960d-471d56d531ee" (UID: "d65dd531-32d4-4276-960d-471d56d531ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:13.065625 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.065601    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:13.068795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.068756    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59" pod "d65dd531-32d4-4276-960d-471d56d531ee" (UID: "d65dd531-32d4-4276-960d-471d56d531ee"). InnerVolumeSpecName "kube-api-access-nqr59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:13.069031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.069004    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca" pod "d65dd531-32d4-4276-960d-471d56d531ee" (UID: "d65dd531-32d4-4276-960d-471d56d531ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:56:13.167687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.167617    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d65dd531-32d4-4276-960d-471d56d531ee-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:13.167687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.167651    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqr59\" (UniqueName: \"kubernetes.io/projected/d65dd531-32d4-4276-960d-471d56d531ee-kube-api-access-nqr59\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:13.803086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.802850    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:13.809119 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.809088    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"
Nov 01 22:56:13.809406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.809105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm" event={"ID":"d65dd531-32d4-4276-960d-471d56d531ee","Type":"ContainerDied","Data":"48db25602d036896b78674ff9bb09b1de236278e3c4369c92fb177fcc8ca528d"}
Nov 01 22:56:13.809547 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:13.809433    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48db25602d036896b78674ff9bb09b1de236278e3c4369c92fb177fcc8ca528d"
Nov 01 22:56:19.028782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:19.028677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:20.610351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:20.610319    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:24.239334 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:24.239288    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:26.993571 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:26.993465    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:29.263205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:29.263151    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:29.448970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:29.448888    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:29.650442 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:29.650357    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:32.901071 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:32.901018    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:33.629903 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.629870    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7128-ec65bd"]
Nov 01 22:56:33.630711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.630689    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d65dd531-32d4-4276-960d-471d56d531ee" containerName="revoke-keys"
Nov 01 22:56:33.630711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.630709    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d65dd531-32d4-4276-960d-471d56d531ee" containerName="revoke-keys"
Nov 01 22:56:33.630995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.630977    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d65dd531-32d4-4276-960d-471d56d531ee" containerName="revoke-keys"
Nov 01 22:56:33.634830 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.634810    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:33.690162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.690136    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-ec65bd"]
Nov 01 22:56:33.730293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.730260    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5sq6\""
Nov 01 22:56:33.813385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.813332    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks8bg\" (UniqueName: \"kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg\") pod \"test-curl-7128-ec65bd\" (UID: \"f2bade7d-6aec-47ee-987a-ec59397c6bfa\") " pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:33.915065 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.914994    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks8bg\" (UniqueName: \"kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg\") pod \"test-curl-7128-ec65bd\" (UID: \"f2bade7d-6aec-47ee-987a-ec59397c6bfa\") " pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:33.971005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:33.970973    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks8bg\" (UniqueName: \"kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg\") pod \"test-curl-7128-ec65bd\" (UID: \"f2bade7d-6aec-47ee-987a-ec59397c6bfa\") " pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:34.247403 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:34.247372    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:34.578330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:34.578244    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:34.645988 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:56:34.645941    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2bade7d_6aec_47ee_987a_ec59397c6bfa.slice/crio-ff5be54e60737a1e82e1f98eadb403829ef9211d59861a4a6cfa924f013836b3 WatchSource:0}: Error finding container ff5be54e60737a1e82e1f98eadb403829ef9211d59861a4a6cfa924f013836b3: Status 404 returned error can't find the container with id ff5be54e60737a1e82e1f98eadb403829ef9211d59861a4a6cfa924f013836b3
Nov 01 22:56:34.646176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:34.646149    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-ec65bd"]
Nov 01 22:56:34.993799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:34.993766    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-ec65bd" event={"ID":"f2bade7d-6aec-47ee-987a-ec59397c6bfa","Type":"ContainerStarted","Data":"115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8"}
Nov 01 22:56:34.994179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:34.993811    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-ec65bd" event={"ID":"f2bade7d-6aec-47ee-987a-ec59397c6bfa","Type":"ContainerStarted","Data":"ff5be54e60737a1e82e1f98eadb403829ef9211d59861a4a6cfa924f013836b3"}
Nov 01 22:56:35.102915 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:35.102871    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7128-ec65bd" podStartSLOduration=2.102856445 podStartE2EDuration="2.102856445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:35.098067375 +0000 UTC m=+1976.415717377" watchObservedRunningTime="2026-11-01 22:56:35.102856445 +0000 UTC m=+1976.420506446"
Nov 01 22:56:36.477455 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.477420    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7128-ec65bd"]
Nov 01 22:56:36.494757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.494715    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7128-ec65bd"]
Nov 01 22:56:36.656110 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.656079    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7128-9804ba"]
Nov 01 22:56:36.656998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.656975    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" containerName="test-curl-7128-ec65bd"
Nov 01 22:56:36.656998 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.656996    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" containerName="test-curl-7128-ec65bd"
Nov 01 22:56:36.657169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.657162    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" containerName="test-curl-7128-ec65bd"
Nov 01 22:56:36.661138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.661116    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:36.713758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.713704    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-9804ba"]
Nov 01 22:56:36.756297 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.756235    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwbl\" (UniqueName: \"kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl\") pod \"test-curl-7128-9804ba\" (UID: \"ae0453b1-c367-4440-a249-f48d2b35f8ac\") " pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:36.858176 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.858146    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwbl\" (UniqueName: \"kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl\") pod \"test-curl-7128-9804ba\" (UID: \"ae0453b1-c367-4440-a249-f48d2b35f8ac\") " pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:36.915782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.915720    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwbl\" (UniqueName: \"kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl\") pod \"test-curl-7128-9804ba\" (UID: \"ae0453b1-c367-4440-a249-f48d2b35f8ac\") " pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:36.975208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:36.975170    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:37.014516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:37.014431    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7128-ec65bd" podUID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" containerName="test-curl-7128-ec65bd" containerID="cri-o://115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8" gracePeriod=2
Nov 01 22:56:37.385280 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:37.385241    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-9804ba"]
Nov 01 22:56:37.386272 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:56:37.386245    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae0453b1_c367_4440_a249_f48d2b35f8ac.slice/crio-c5578aa8e18476894b99dea7736734bfb256ffe3d65336116bd4283639bd40ab WatchSource:0}: Error finding container c5578aa8e18476894b99dea7736734bfb256ffe3d65336116bd4283639bd40ab: Status 404 returned error can't find the container with id c5578aa8e18476894b99dea7736734bfb256ffe3d65336116bd4283639bd40ab
Nov 01 22:56:38.024788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:38.024692    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-9804ba" event={"ID":"ae0453b1-c367-4440-a249-f48d2b35f8ac","Type":"ContainerStarted","Data":"9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7"}
Nov 01 22:56:38.024788 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:38.024782    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-9804ba" event={"ID":"ae0453b1-c367-4440-a249-f48d2b35f8ac","Type":"ContainerStarted","Data":"c5578aa8e18476894b99dea7736734bfb256ffe3d65336116bd4283639bd40ab"}
Nov 01 22:56:38.131307 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:38.131248    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7128-9804ba" podStartSLOduration=2.131235051 podStartE2EDuration="2.131235051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:38.129068683 +0000 UTC m=+1979.446718683" watchObservedRunningTime="2026-11-01 22:56:38.131235051 +0000 UTC m=+1979.448885052"
Nov 01 22:56:38.823100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:38.823064    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:39.301989 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.301958    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:39.376860 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.376821    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:39.416025 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.413477    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg\" (UniqueName: \"kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg\") pod \"f2bade7d-6aec-47ee-987a-ec59397c6bfa\" (UID: \"f2bade7d-6aec-47ee-987a-ec59397c6bfa\") "
Nov 01 22:56:39.440039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.437193    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg" pod "f2bade7d-6aec-47ee-987a-ec59397c6bfa" (UID: "f2bade7d-6aec-47ee-987a-ec59397c6bfa"). InnerVolumeSpecName "kube-api-access-ks8bg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:39.516659 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.516620    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks8bg\" (UniqueName: \"kubernetes.io/projected/f2bade7d-6aec-47ee-987a-ec59397c6bfa-kube-api-access-ks8bg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:39.565513 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.565452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:39.621666 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.621628    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7128-9804ba"]
Nov 01 22:56:39.639673 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.639637    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7128-9804ba"]
Nov 01 22:56:39.766190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.766151    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7128-ec9621"]
Nov 01 22:56:39.766971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.766944    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" containerName="test-curl-7128-9804ba"
Nov 01 22:56:39.766971 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.766970    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" containerName="test-curl-7128-9804ba"
Nov 01 22:56:39.767167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.767154    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" containerName="test-curl-7128-9804ba"
Nov 01 22:56:39.767679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.767626    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:39.771040 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.771018    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:39.821318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.821290    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-ec9621"]
Nov 01 22:56:39.825928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.825432    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww64d\" (UniqueName: \"kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d\") pod \"test-curl-7128-ec9621\" (UID: \"2381d606-6642-4d14-93d9-6fa7f1afe2e5\") " pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:39.926820 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.926791    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww64d\" (UniqueName: \"kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d\") pod \"test-curl-7128-ec9621\" (UID: \"2381d606-6642-4d14-93d9-6fa7f1afe2e5\") " pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:39.985893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:39.985858    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww64d\" (UniqueName: \"kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d\") pod \"test-curl-7128-ec9621\" (UID: \"2381d606-6642-4d14-93d9-6fa7f1afe2e5\") " pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:40.046572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.046496    4531 generic.go:356] "Generic (PLEG): container finished" podID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" containerID="115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8" exitCode=137
Nov 01 22:56:40.046572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.046557    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:40.046776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.046606    4531 scope.go:122] "RemoveContainer" containerID="115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8"
Nov 01 22:56:40.046776 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.046638    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7128-9804ba" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" containerName="test-curl-7128-9804ba" containerID="cri-o://9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7" gracePeriod=2
Nov 01 22:56:40.061812 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.061791    4531 scope.go:122] "RemoveContainer" containerID="115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8"
Nov 01 22:56:40.062104 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:40.062077    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8\": container with ID starting with 115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8 not found: ID does not exist" containerID="115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8"
Nov 01 22:56:40.062199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.062112    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8"} err="failed to get container status \"115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8\": rpc error: code = NotFound desc = could not find container \"115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8\": container with ID starting with 115f261edf6fd74cfd2d4eb12c86a1905d4302ac8a4b2ebe83597ea99aff20b8 not found: ID does not exist"
Nov 01 22:56:40.084657 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.084630    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:40.094392 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:40.094351    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-9804ba\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:40.142007 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:40.141973    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-9804ba\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:40.189605 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:40.189509    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-ec65bd\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" pod="opendatahub/test-curl-7128-ec65bd"
Nov 01 22:56:40.468313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.468228    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-ec9621"]
Nov 01 22:56:40.471178 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:56:40.471151    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2381d606_6642_4d14_93d9_6fa7f1afe2e5.slice/crio-4db61ddb3f6196467c7993230cf2b8e1ab65554606d12c60ec3c27a7aac80acb WatchSource:0}: Error finding container 4db61ddb3f6196467c7993230cf2b8e1ab65554606d12c60ec3c27a7aac80acb: Status 404 returned error can't find the container with id 4db61ddb3f6196467c7993230cf2b8e1ab65554606d12c60ec3c27a7aac80acb
Nov 01 22:56:40.633193 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.633168    4531 scope.go:122] "RemoveContainer" containerID="d3dc8da8cd8660c619a191e1c481c2bdc97aa8302e1de62b0794e2710f2382f1"
Nov 01 22:56:40.643287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.643267    4531 scope.go:122] "RemoveContainer" containerID="7ee54e58137001b084128c019054a72eaa805445560ef0039c7d7521e135d802"
Nov 01 22:56:40.818738 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:40.818660    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2bade7d-6aec-47ee-987a-ec59397c6bfa" path="/var/lib/kubelet/pods/f2bade7d-6aec-47ee-987a-ec59397c6bfa/volumes"
Nov 01 22:56:41.056263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:41.056232    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-ec9621" event={"ID":"2381d606-6642-4d14-93d9-6fa7f1afe2e5","Type":"ContainerStarted","Data":"e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a"}
Nov 01 22:56:41.056430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:41.056277    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-ec9621" event={"ID":"2381d606-6642-4d14-93d9-6fa7f1afe2e5","Type":"ContainerStarted","Data":"4db61ddb3f6196467c7993230cf2b8e1ab65554606d12c60ec3c27a7aac80acb"}
Nov 01 22:56:41.164155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:41.164064    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7128-ec9621" podStartSLOduration=2.164051033 podStartE2EDuration="2.164051033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:41.16187403 +0000 UTC m=+1982.479524030" watchObservedRunningTime="2026-11-01 22:56:41.164051033 +0000 UTC m=+1982.481701033"
Nov 01 22:56:42.848772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:42.848319    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:42.861748 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:42.858970    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:56:42.882749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:42.879953    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:42.907881 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:42.903352    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl\" (UniqueName: \"kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl\") pod \"ae0453b1-c367-4440-a249-f48d2b35f8ac\" (UID: \"ae0453b1-c367-4440-a249-f48d2b35f8ac\") "
Nov 01 22:56:42.928748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:42.918188    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl" pod "ae0453b1-c367-4440-a249-f48d2b35f8ac" (UID: "ae0453b1-c367-4440-a249-f48d2b35f8ac"). InnerVolumeSpecName "kube-api-access-cfwbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:43.014758 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.012012    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwbl\" (UniqueName: \"kubernetes.io/projected/ae0453b1-c367-4440-a249-f48d2b35f8ac-kube-api-access-cfwbl\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:43.144156 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.143530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:43.171200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.171167    4531 generic.go:356] "Generic (PLEG): container finished" podID="ae0453b1-c367-4440-a249-f48d2b35f8ac" containerID="9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7" exitCode=137
Nov 01 22:56:43.171557 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.171453    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-9804ba"
Nov 01 22:56:43.193750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.171483    4531 scope.go:122] "RemoveContainer" containerID="9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7"
Nov 01 22:56:43.333804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.331783    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7128-ec9621"]
Nov 01 22:56:43.333804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.331978    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7128-ec9621" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" containerName="test-curl-7128-ec9621" containerID="cri-o://e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a" gracePeriod=2
Nov 01 22:56:43.335057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.334951    4531 scope.go:122] "RemoveContainer" containerID="9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7"
Nov 01 22:56:43.335721 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:43.335644    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7\": container with ID starting with 9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7 not found: ID does not exist" containerID="9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7"
Nov 01 22:56:43.335721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.335681    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7"} err="failed to get container status \"9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7\": rpc error: code = NotFound desc = could not find container \"9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7\": container with ID starting with 9fc446b81900f1cf9038be71fad0d8dca689934d0312358a577fd82ebf2401b7 not found: ID does not exist"
Nov 01 22:56:43.338882 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.338860    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7128-ec9621"]
Nov 01 22:56:43.430033 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:43.429940    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-ec9621\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:43.551367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.538905    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7128-5e8702"]
Nov 01 22:56:43.551367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.540209    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" containerName="test-curl-7128-ec9621"
Nov 01 22:56:43.551367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.540232    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" containerName="test-curl-7128-ec9621"
Nov 01 22:56:43.551367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.540451    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" containerName="test-curl-7128-ec9621"
Nov 01 22:56:43.551367 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.546937    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:43.583749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.578917    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-5e8702"]
Nov 01 22:56:43.676271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.676181    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mj8z\" (UniqueName: \"kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z\") pod \"test-curl-7128-5e8702\" (UID: \"734caa23-dd30-4741-a710-cb9092c9ca64\") " pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:43.700185 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:43.700152    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-ec9621\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:43.777089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.777052    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mj8z\" (UniqueName: \"kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z\") pod \"test-curl-7128-5e8702\" (UID: \"734caa23-dd30-4741-a710-cb9092c9ca64\") " pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:43.833761 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.833703    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mj8z\" (UniqueName: \"kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z\") pod \"test-curl-7128-5e8702\" (UID: \"734caa23-dd30-4741-a710-cb9092c9ca64\") " pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:43.884061 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.884031    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:43.981157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:43.981052    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:44.187763 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:44.187696    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:44.298660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:44.298626    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7128-5e8702"]
Nov 01 22:56:44.404293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:44.404258    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:44.653215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:44.653090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:44.821344 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:44.821312    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0453b1-c367-4440-a249-f48d2b35f8ac" path="/var/lib/kubelet/pods/ae0453b1-c367-4440-a249-f48d2b35f8ac/volumes"
Nov 01 22:56:45.197988 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.197950    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-5e8702" event={"ID":"734caa23-dd30-4741-a710-cb9092c9ca64","Type":"ContainerStarted","Data":"a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a"}
Nov 01 22:56:45.198440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.197999    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7128-5e8702" event={"ID":"734caa23-dd30-4741-a710-cb9092c9ca64","Type":"ContainerStarted","Data":"001d20dcd45f7c78829ac2db25847db403d6ee183a527a4cad8b9f90a27aefbb"}
Nov 01 22:56:45.312766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.312693    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7128-5e8702" podStartSLOduration=2.312677659 podStartE2EDuration="2.312677659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:45.310350932 +0000 UTC m=+1986.628000925" watchObservedRunningTime="2026-11-01 22:56:45.312677659 +0000 UTC m=+1986.630327661"
Nov 01 22:56:45.595201 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.595175    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:45.720401 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.720368    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d\" (UniqueName: \"kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d\") pod \"2381d606-6642-4d14-93d9-6fa7f1afe2e5\" (UID: \"2381d606-6642-4d14-93d9-6fa7f1afe2e5\") "
Nov 01 22:56:45.722512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.722485    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d" pod "2381d606-6642-4d14-93d9-6fa7f1afe2e5" (UID: "2381d606-6642-4d14-93d9-6fa7f1afe2e5"). InnerVolumeSpecName "kube-api-access-ww64d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:45.824441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:45.824409    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww64d\" (UniqueName: \"kubernetes.io/projected/2381d606-6642-4d14-93d9-6fa7f1afe2e5-kube-api-access-ww64d\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:46.207821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.207783    4531 generic.go:356] "Generic (PLEG): container finished" podID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" containerID="e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a" exitCode=137
Nov 01 22:56:46.208271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.207847    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:46.208271 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.207889    4531 scope.go:122] "RemoveContainer" containerID="e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a"
Nov 01 22:56:46.219670 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.219653    4531 scope.go:122] "RemoveContainer" containerID="e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a"
Nov 01 22:56:46.219983 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:46.219934    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a\": container with ID starting with e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a not found: ID does not exist" containerID="e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a"
Nov 01 22:56:46.220085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.219995    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a"} err="failed to get container status \"e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a\": rpc error: code = NotFound desc = could not find container \"e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a\": container with ID starting with e2c3f26918d0d4a355706d6d59d074f9aae9205c8393049827a7a293f526c51a not found: ID does not exist"
Nov 01 22:56:46.264304 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:46.264272    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-ec9621\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" pod="opendatahub/test-curl-7128-ec9621"
Nov 01 22:56:46.732422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.732367    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7128-5e8702"]
Nov 01 22:56:46.738290 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.738261    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7128-5e8702"]
Nov 01 22:56:46.819215 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.819187    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2381d606-6642-4d14-93d9-6fa7f1afe2e5" path="/var/lib/kubelet/pods/2381d606-6642-4d14-93d9-6fa7f1afe2e5/volumes"
Nov 01 22:56:46.997610 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.997536    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"]
Nov 01 22:56:46.997797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:46.997739    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy" containerID="cri-o://f1920d96f2e935e4e297a08de0528ecc5cd09ce533fcd34bc604640ee84eb4f9" gracePeriod=30
Nov 01 22:56:47.189180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.189147    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"]
Nov 01 22:56:47.189385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.189358    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" containerID="cri-o://ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59" gracePeriod=30
Nov 01 22:56:47.195161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.195139    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"]
Nov 01 22:56:47.195336 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.195316    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy" containerID="cri-o://44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689" gracePeriod=30
Nov 01 22:56:47.222250 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.222209    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7128-5e8702" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" containerName="test-curl-7128-5e8702" containerID="cri-o://a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a" gracePeriod=2
Nov 01 22:56:47.328436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.328358    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"]
Nov 01 22:56:47.328619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.328594    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" containerID="cri-o://3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d" gracePeriod=30
Nov 01 22:56:47.395846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:47.395814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:48.179816 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:48.179769    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:56:48.179964 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:48.179874    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:56:48.303211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:48.303179    4531 patch_prober.go:29] interesting pod/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:56:48.303622 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:48.303239    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:56:49.453097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:49.453067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:49.530498 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:49.530463    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:49.612027 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:49.611999    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z\" (UniqueName: \"kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z\") pod \"734caa23-dd30-4741-a710-cb9092c9ca64\" (UID: \"734caa23-dd30-4741-a710-cb9092c9ca64\") "
Nov 01 22:56:49.614031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:49.614004    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z" pod "734caa23-dd30-4741-a710-cb9092c9ca64" (UID: "734caa23-dd30-4741-a710-cb9092c9ca64"). InnerVolumeSpecName "kube-api-access-9mj8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:49.714286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:49.714262    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mj8z\" (UniqueName: \"kubernetes.io/projected/734caa23-dd30-4741-a710-cb9092c9ca64-kube-api-access-9mj8z\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:49.961750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:49.958677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:50.149443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.149397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:50.251538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.251453    4531 generic.go:356] "Generic (PLEG): container finished" podID="734caa23-dd30-4741-a710-cb9092c9ca64" containerID="a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a" exitCode=137
Nov 01 22:56:50.251538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.251515    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:50.251538 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.251540    4531 scope.go:122] "RemoveContainer" containerID="a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a"
Nov 01 22:56:50.262868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.262847    4531 scope.go:122] "RemoveContainer" containerID="a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a"
Nov 01 22:56:50.263432 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:50.263411    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a\": container with ID starting with a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a not found: ID does not exist" containerID="a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a"
Nov 01 22:56:50.263553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.263443    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a"} err="failed to get container status \"a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a\": rpc error: code = NotFound desc = could not find container \"a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a\": container with ID starting with a52ebf3c7b069a7d47b0c9749df9abfcad6bfffdcdf6c9993e33cd65f9ca0b6a not found: ID does not exist"
Nov 01 22:56:50.284699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.284644    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"]
Nov 01 22:56:50.284926 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.284900    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" containerID="cri-o://c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405" gracePeriod=30
Nov 01 22:56:50.325122 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:50.325084    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-5e8702\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:50.372663 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:50.372622    4531 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7128-5e8702\" is forbidden: User \"system:node:ip-10-0-130-144.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-144.us-west-2.compute.internal' and this object" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" pod="opendatahub/test-curl-7128-5e8702"
Nov 01 22:56:50.822335 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:50.822301    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" path="/var/lib/kubelet/pods/734caa23-dd30-4741-a710-cb9092c9ca64/volumes"
Nov 01 22:56:52.237475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.237183    4531 patch_prober.go:29] interesting pod/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:56:52.237475 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.237239    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:56:52.271753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.271684    4531 generic.go:356] "Generic (PLEG): container finished" podID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerID="f1920d96f2e935e4e297a08de0528ecc5cd09ce533fcd34bc604640ee84eb4f9" exitCode=0
Nov 01 22:56:52.271867 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.271775    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" event={"ID":"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7","Type":"ContainerDied","Data":"f1920d96f2e935e4e297a08de0528ecc5cd09ce533fcd34bc604640ee84eb4f9"}
Nov 01 22:56:52.421388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.420891    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:56:52.552138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.552066    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:56:52.569789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.569535    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.569789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.569630    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.569789 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.569685    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.570085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.569873    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.570085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.569943    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.570085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.569989    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.570085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.570022    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.570085 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.570056    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.570382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.570104    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data\") pod \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\" (UID: \"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7\") "
Nov 01 22:56:52.579710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.571264    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.579710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.571504    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.579710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.571852    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.579710 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.572067    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.584406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.581508    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "kube-api-access-pntrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.584406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.581654    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.584406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.582046    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:52.589868 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.589571    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:52.593219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.591578    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token" pod "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" (UID: "012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671437    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671496    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671544    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671602    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671744    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671823    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.671870    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672024    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672076    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy\") pod \"0f006c60-0006-4d13-a623-2d4cb888b717\" (UID: \"0f006c60-0006-4d13-a623-2d4cb888b717\") "
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672850    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672876    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672890    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672904    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672918    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pntrq\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-kube-api-access-pntrq\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672933    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672947    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672960    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.672974    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.673788    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.674503    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.674795    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.675038    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.684248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.675516    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:52.688760 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.688697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.689892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.689721    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "kube-api-access-c57vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.691584 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.691528    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:52.692226 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.692202    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy" pod "0f006c60-0006-4d13-a623-2d4cb888b717" (UID: "0f006c60-0006-4d13-a623-2d4cb888b717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.703568 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.703547    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774220    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774255    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c57vj\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-kube-api-access-c57vj\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774271    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f006c60-0006-4d13-a623-2d4cb888b717-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774283    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f006c60-0006-4d13-a623-2d4cb888b717-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774297    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f006c60-0006-4d13-a623-2d4cb888b717-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774310    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774323    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774335    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.774969 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.774349    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f006c60-0006-4d13-a623-2d4cb888b717-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.876412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876373    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876443    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876586 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876538    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876605    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876716    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876878 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876772    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.876999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.876968    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.877075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.877047    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:52.877130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.877075    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo\") pod \"2f1f5981-2381-4f38-9fcb-38fd848a8767\" (UID: \"2f1f5981-2381-4f38-9fcb-38fd848a8767\") "
Nov 01 22:56:52.877347 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.877324    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.877910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.877884    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.878009 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.877913    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2f1f5981-2381-4f38-9fcb-38fd848a8767-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.879932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.879900    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.881596 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.881570    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.881842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.881819    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.883035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.883014    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.906651 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.906624    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:52.909485 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.909456    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "kube-api-access-zcn4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:52.910026 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.910006    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo" pod "2f1f5981-2381-4f38-9fcb-38fd848a8767" (UID: "2f1f5981-2381-4f38-9fcb-38fd848a8767"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:52.919850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.919832    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:56:52.981887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981828    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.981887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981885    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.982147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981902    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.982147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981916    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcn4g\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-kube-api-access-zcn4g\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.982147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981931    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.982147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981947    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:52.982147 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:52.981959    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2f1f5981-2381-4f38-9fcb-38fd848a8767-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.082974 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.082875    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083059    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083123    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083152    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083196    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083269    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083394    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083432    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083480 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083472    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data\") pod \"647cd028-ce48-4e6a-930d-34fc3c8a6342\" (UID: \"647cd028-ce48-4e6a-930d-34fc3c8a6342\") "
Nov 01 22:56:53.083999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.083971    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.084869 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.084846    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.085377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.085354    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:53.088164 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.088136    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.088385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.088361    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.096716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.096681    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:53.097435 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.097398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:53.097684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.097647    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "kube-api-access-6hmf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:53.097794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.097717    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy" pod "647cd028-ce48-4e6a-930d-34fc3c8a6342" (UID: "647cd028-ce48-4e6a-930d-34fc3c8a6342"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:53.185601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185560    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185593    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185606    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185618    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185631    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmf5\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-kube-api-access-6hmf5\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185646    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185658    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/647cd028-ce48-4e6a-930d-34fc3c8a6342-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185671    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/647cd028-ce48-4e6a-930d-34fc3c8a6342-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.185842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.185682    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/647cd028-ce48-4e6a-930d-34fc3c8a6342-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:53.282753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.282692    4531 generic.go:356] "Generic (PLEG): container finished" podID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerID="44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689" exitCode=0
Nov 01 22:56:53.283207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.282788    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" event={"ID":"647cd028-ce48-4e6a-930d-34fc3c8a6342","Type":"ContainerDied","Data":"44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689"}
Nov 01 22:56:53.283207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.282813    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"
Nov 01 22:56:53.283207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.282877    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt" event={"ID":"647cd028-ce48-4e6a-930d-34fc3c8a6342","Type":"ContainerDied","Data":"8a22aead73c12dc7d1e27d1158798675102193d216e6cfd8eaa073015b10a50e"}
Nov 01 22:56:53.283207 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.282911    4531 scope.go:122] "RemoveContainer" containerID="44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689"
Nov 01 22:56:53.284650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.284623    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh" event={"ID":"012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7","Type":"ContainerDied","Data":"d2ab34aa1ae4c23a35b3b2f62b5e69d5b136e0c86b949b144cfb4c4206fc6bbb"}
Nov 01 22:56:53.284778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.284650    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"
Nov 01 22:56:53.286550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.286516    4531 generic.go:356] "Generic (PLEG): container finished" podID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerID="3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d" exitCode=0
Nov 01 22:56:53.286637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.286592    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"
Nov 01 22:56:53.286722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.286702    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" event={"ID":"2f1f5981-2381-4f38-9fcb-38fd848a8767","Type":"ContainerDied","Data":"3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d"}
Nov 01 22:56:53.286800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.286773    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h" event={"ID":"2f1f5981-2381-4f38-9fcb-38fd848a8767","Type":"ContainerDied","Data":"c996b0c28e2de9ac16ba0cdcdabb9f918ab801fe90bf5bc68d20585765de85a0"}
Nov 01 22:56:53.291995 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.291969    4531 generic.go:356] "Generic (PLEG): container finished" podID="0f006c60-0006-4d13-a623-2d4cb888b717" containerID="ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59" exitCode=0
Nov 01 22:56:53.292109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.292023    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" event={"ID":"0f006c60-0006-4d13-a623-2d4cb888b717","Type":"ContainerDied","Data":"ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59"}
Nov 01 22:56:53.292109 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.292056    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"
Nov 01 22:56:53.292234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.292059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5" event={"ID":"0f006c60-0006-4d13-a623-2d4cb888b717","Type":"ContainerDied","Data":"3176ea56753009dfdeeb75837860451cd35203d7693023a367d67ae3dd0ed143"}
Nov 01 22:56:53.296380 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.296353    4531 scope.go:122] "RemoveContainer" containerID="44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689"
Nov 01 22:56:53.296773 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:53.296714    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689\": container with ID starting with 44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689 not found: ID does not exist" containerID="44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689"
Nov 01 22:56:53.296773 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.296760    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689"} err="failed to get container status \"44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689\": rpc error: code = NotFound desc = could not find container \"44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689\": container with ID starting with 44529b7a77018e063d5ba55a54f5bbb880404ef408095967c59c17cd38153689 not found: ID does not exist"
Nov 01 22:56:53.296949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.296785    4531 scope.go:122] "RemoveContainer" containerID="f1920d96f2e935e4e297a08de0528ecc5cd09ce533fcd34bc604640ee84eb4f9"
Nov 01 22:56:53.309346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.309327    4531 scope.go:122] "RemoveContainer" containerID="3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d"
Nov 01 22:56:53.319438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.319412    4531 scope.go:122] "RemoveContainer" containerID="3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d"
Nov 01 22:56:53.319702 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:53.319682    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d\": container with ID starting with 3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d not found: ID does not exist" containerID="3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d"
Nov 01 22:56:53.319794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.319708    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d"} err="failed to get container status \"3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d\": rpc error: code = NotFound desc = could not find container \"3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d\": container with ID starting with 3076dacadd3468205ad3b42a692cd894619196118269f2474a705dfb31aa656d not found: ID does not exist"
Nov 01 22:56:53.319794 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.319755    4531 scope.go:122] "RemoveContainer" containerID="ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59"
Nov 01 22:56:53.329004 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.328987    4531 scope.go:122] "RemoveContainer" containerID="ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59"
Nov 01 22:56:53.329307 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:53.329283    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59\": container with ID starting with ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59 not found: ID does not exist" containerID="ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59"
Nov 01 22:56:53.329389 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.329314    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59"} err="failed to get container status \"ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59\": rpc error: code = NotFound desc = could not find container \"ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59\": container with ID starting with ed4c50dd95fa5fcd338e8108c22f5b4d1fb64444ad51687142563050086a8b59 not found: ID does not exist"
Nov 01 22:56:53.440817 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.440751    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"]
Nov 01 22:56:53.486736 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.486707    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-9zpmh"]
Nov 01 22:56:53.552406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.552378    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"]
Nov 01 22:56:53.559312 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.559286    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-rd5mt"]
Nov 01 22:56:53.661502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.661473    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"]
Nov 01 22:56:53.666691 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.666667    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-cwmd5"]
Nov 01 22:56:53.769174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.769143    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"]
Nov 01 22:56:53.773502 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:53.773479    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-6cs7h"]
Nov 01 22:56:54.116460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.116376    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:54.321953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.321920    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:54.550749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.546962    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:54.820163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.820088    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" path="/var/lib/kubelet/pods/012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7/volumes"
Nov 01 22:56:54.820607 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.820591    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" path="/var/lib/kubelet/pods/0f006c60-0006-4d13-a623-2d4cb888b717/volumes"
Nov 01 22:56:54.821115 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.821099    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" path="/var/lib/kubelet/pods/2f1f5981-2381-4f38-9fcb-38fd848a8767/volumes"
Nov 01 22:56:54.821910 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:54.821895    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" path="/var/lib/kubelet/pods/647cd028-ce48-4e6a-930d-34fc3c8a6342/volumes"
Nov 01 22:56:55.586992 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.586967    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:56:55.732911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.732887    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.732930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.732988    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733036    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733067 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733062    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733281 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733235    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733300    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733289    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.733338 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733332    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733343    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.733495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733420    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket\") pod \"8ba4fbea-c951-4772-849f-80f6eb700609\" (UID: \"8ba4fbea-c951-4772-849f-80f6eb700609\") "
Nov 01 22:56:55.733853 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733591    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.733853 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733751    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:55.733853 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.733721    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.734257 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.734236    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ba4fbea-c951-4772-849f-80f6eb700609-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.734303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.734266    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.734303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.734284    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.734392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.734299    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.734392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.734334    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.735656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.735628    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "kube-api-access-4szzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:55.735782 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.735759    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:55.735853 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.735830    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:56:55.735951 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.735931    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token" pod "8ba4fbea-c951-4772-849f-80f6eb700609" (UID: "8ba4fbea-c951-4772-849f-80f6eb700609"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:55.829746 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.829658    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"]
Nov 01 22:56:55.829916 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.829897    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" podUID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" containerName="discovery" containerID="cri-o://d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5" gracePeriod=30
Nov 01 22:56:55.835639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.835613    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ba4fbea-c951-4772-849f-80f6eb700609-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.835639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.835639    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4szzv\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-kube-api-access-4szzv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.835810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.835651    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ba4fbea-c951-4772-849f-80f6eb700609-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:55.835810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:55.835660    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ba4fbea-c951-4772-849f-80f6eb700609-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.128030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.128003    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:56:56.143356 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143315    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.143522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143501    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.143599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143573    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.143671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143598    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.143671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143625    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.143813 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143797    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.143934 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143920    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token\") pod \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\" (UID: \"04f8d576-38b7-4f1e-aaa7-6b6434ade817\") "
Nov 01 22:56:56.144007 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.143978    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:56:56.144772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.144711    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-ca-configmap\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.146187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.146165    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "kube-api-access-kd4xg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:56.146313 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.146279    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:56:56.146419 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.146322    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:56:56.146512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.146497    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:56:56.146641 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.146626    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:56:56.146702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.146662    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs" pod "04f8d576-38b7-4f1e-aaa7-6b6434ade817" (UID: "04f8d576-38b7-4f1e-aaa7-6b6434ade817"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:56:56.247075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.247046    4531 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-cacerts\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.247256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.247078    4531 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/04f8d576-38b7-4f1e-aaa7-6b6434ade817-local-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.247256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.247097    4531 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-csr-dns-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.247256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.247110    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.247256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.247125    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd4xg\" (UniqueName: \"kubernetes.io/projected/04f8d576-38b7-4f1e-aaa7-6b6434ade817-kube-api-access-kd4xg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.247256 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.247139    4531 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04f8d576-38b7-4f1e-aaa7-6b6434ade817-istio-kubeconfig\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:56:56.327293 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.327229    4531 generic.go:356] "Generic (PLEG): container finished" podID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" containerID="d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5" exitCode=0
Nov 01 22:56:56.328008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.327290    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"
Nov 01 22:56:56.328008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.327359    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" event={"ID":"04f8d576-38b7-4f1e-aaa7-6b6434ade817","Type":"ContainerDied","Data":"d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5"}
Nov 01 22:56:56.328008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.327426    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j" event={"ID":"04f8d576-38b7-4f1e-aaa7-6b6434ade817","Type":"ContainerDied","Data":"86be1d2a35dbf83d8ec0667ab1a7ce5a36a21b6919824084871d3aab6290ddf4"}
Nov 01 22:56:56.328008 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.327458    4531 scope.go:122] "RemoveContainer" containerID="d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5"
Nov 01 22:56:56.330174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.330064    4531 generic.go:356] "Generic (PLEG): container finished" podID="8ba4fbea-c951-4772-849f-80f6eb700609" containerID="c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405" exitCode=0
Nov 01 22:56:56.330174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.330100    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" event={"ID":"8ba4fbea-c951-4772-849f-80f6eb700609","Type":"ContainerDied","Data":"c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405"}
Nov 01 22:56:56.330174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.330120    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"
Nov 01 22:56:56.330174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.330135    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn" event={"ID":"8ba4fbea-c951-4772-849f-80f6eb700609","Type":"ContainerDied","Data":"6e2a83078df8d483801346ef72205a713cc4e252b124cef5c13979f9c0479a32"}
Nov 01 22:56:56.345699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.345675    4531 scope.go:122] "RemoveContainer" containerID="d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5"
Nov 01 22:56:56.346082 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:56.346057    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5\": container with ID starting with d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5 not found: ID does not exist" containerID="d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5"
Nov 01 22:56:56.346160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.346089    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5"} err="failed to get container status \"d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5\": rpc error: code = NotFound desc = could not find container \"d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5\": container with ID starting with d717f9fa134d09114985ceb24e1def7e53d73b0a3c1deff929f394ef4e83add5 not found: ID does not exist"
Nov 01 22:56:56.346160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.346105    4531 scope.go:122] "RemoveContainer" containerID="c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405"
Nov 01 22:56:56.362553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.361097    4531 scope.go:122] "RemoveContainer" containerID="c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405"
Nov 01 22:56:56.362553 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:56:56.361519    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405\": container with ID starting with c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405 not found: ID does not exist" containerID="c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405"
Nov 01 22:56:56.362553 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.361555    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405"} err="failed to get container status \"c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405\": rpc error: code = NotFound desc = could not find container \"c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405\": container with ID starting with c8a8d785cb2efc1a1478442d20a1e676cb2c10ae1420838350e6e185f072c405 not found: ID does not exist"
Nov 01 22:56:56.487208 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.487173    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"]
Nov 01 22:56:56.492143 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.492114    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-bqfwn"]
Nov 01 22:56:56.595815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.595785    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"]
Nov 01 22:56:56.600779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.600721    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gzf6j"]
Nov 01 22:56:56.827100 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.827020    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" path="/var/lib/kubelet/pods/04f8d576-38b7-4f1e-aaa7-6b6434ade817/volumes"
Nov 01 22:56:56.827666 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:56.827648    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" path="/var/lib/kubelet/pods/8ba4fbea-c951-4772-849f-80f6eb700609/volumes"
Nov 01 22:56:57.139136 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.139044    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:57.535749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.535289    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:57.732755 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.729677    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:56:57.832378 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.832293    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"]
Nov 01 22:56:57.833424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833397    4531 cpu_manager.go:414] "containerMap: removing container" podUID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" containerName="discovery"
Nov 01 22:56:57.833424 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833423    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" containerName="discovery"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833440    4531 cpu_manager.go:414] "containerMap: removing container" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833448    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833477    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833485    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833514    4531 cpu_manager.go:414] "containerMap: removing container" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833521    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833552    4531 cpu_manager.go:414] "containerMap: removing container" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" containerName="test-curl-7128-5e8702"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833559    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" containerName="test-curl-7128-5e8702"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833575    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833582    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy"
Nov 01 22:56:57.833594 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833598    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833605    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833772    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="647cd028-ce48-4e6a-930d-34fc3c8a6342" containerName="istio-proxy"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833794    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="734caa23-dd30-4741-a710-cb9092c9ca64" containerName="test-curl-7128-5e8702"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833805    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f006c60-0006-4d13-a623-2d4cb888b717" containerName="istio-proxy"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833815    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f1f5981-2381-4f38-9fcb-38fd848a8767" containerName="istio-proxy"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833826    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="04f8d576-38b7-4f1e-aaa7-6b6434ade817" containerName="discovery"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833835    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="012ee2d9-ed1e-4fab-ad3f-24d0eb3fbfd7" containerName="istio-proxy"
Nov 01 22:56:57.833967 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.833846    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba4fbea-c951-4772-849f-80f6eb700609" containerName="istio-proxy"
Nov 01 22:56:57.840112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.840086    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:57.889752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.889489    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"]
Nov 01 22:56:57.929291 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.929267    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d68bab\""
Nov 01 22:56:57.931844 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.931814    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:57.932128 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.932104    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj8h9\" (UniqueName: \"kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9\") pod \"payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:57.995749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:57.995555    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"]
Nov 01 22:56:58.018751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.001267    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.041795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.037471    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9llw\" (UniqueName: \"kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw\") pod \"payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") " pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.041795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.037585    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj8h9\" (UniqueName: \"kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9\") pod \"payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:58.041795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.037667    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:58.041795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.037709    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") " pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.041795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.038621    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:58.082479 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.082399    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d68bab-dockercfg-knsxf\""
Nov 01 22:56:58.083092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.083065    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"]
Nov 01 22:56:58.106097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.105059    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj8h9\" (UniqueName: \"kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9\") pod \"payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:58.139997 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.139958    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9llw\" (UniqueName: \"kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw\") pod \"payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") " pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.140183 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.140164    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") " pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.140976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.140951    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") " pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.158569 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.158543    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:58.200825 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.200790    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9llw\" (UniqueName: \"kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw\") pod \"payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") " pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.321094 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.321060    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:58.561838 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.561794    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"]
Nov 01 22:56:58.565324 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:56:58.565298    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fa3111e_1565_4bcd_913a_c3161e31c140.slice/crio-08d735db855ebe5e107d6e7dd05b2c9c093cda3bb6632f712489ffb37af9c935 WatchSource:0}: Error finding container 08d735db855ebe5e107d6e7dd05b2c9c093cda3bb6632f712489ffb37af9c935: Status 404 returned error can't find the container with id 08d735db855ebe5e107d6e7dd05b2c9c093cda3bb6632f712489ffb37af9c935
Nov 01 22:56:58.708605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:58.708578    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"]
Nov 01 22:56:58.710775 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:56:58.710707    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8a2634_02cf_48b2_9679_037cb4a2783d.slice/crio-96bbbfec8f78b3abc29da28269036e9b7429d83c19c5cf78221973734e707081 WatchSource:0}: Error finding container 96bbbfec8f78b3abc29da28269036e9b7429d83c19c5cf78221973734e707081: Status 404 returned error can't find the container with id 96bbbfec8f78b3abc29da28269036e9b7429d83c19c5cf78221973734e707081
Nov 01 22:56:59.362619 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.362581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj" event={"ID":"1fa3111e-1565-4bcd-913a-c3161e31c140","Type":"ContainerStarted","Data":"17e1fbfccee397b1973740ee21c2de45efe8e60c340b807828ee8eddfa6a1393"}
Nov 01 22:56:59.363157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.362677    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj" event={"ID":"1fa3111e-1565-4bcd-913a-c3161e31c140","Type":"ContainerStarted","Data":"08d735db855ebe5e107d6e7dd05b2c9c093cda3bb6632f712489ffb37af9c935"}
Nov 01 22:56:59.363157 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.362707    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:56:59.364661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.364633    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5" event={"ID":"2b8a2634-02cf-48b2-9679-037cb4a2783d","Type":"ContainerStarted","Data":"5dc3ec3c71d4e86ff43dbb91e52661d2d0eae9bc9a71a8378b064bbfcdf900b1"}
Nov 01 22:56:59.364843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.364680    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5" event={"ID":"2b8a2634-02cf-48b2-9679-037cb4a2783d","Type":"ContainerStarted","Data":"96bbbfec8f78b3abc29da28269036e9b7429d83c19c5cf78221973734e707081"}
Nov 01 22:56:59.364843 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.364709    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:56:59.470754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.470692    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj" podStartSLOduration=2.470679983 podStartE2EDuration="2.470679983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:59.468464506 +0000 UTC m=+2000.786114504" watchObservedRunningTime="2026-11-01 22:56:59.470679983 +0000 UTC m=+2000.788329983"
Nov 01 22:56:59.577262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.577208    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5" podStartSLOduration=2.577190626 podStartE2EDuration="2.577190626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:56:59.576244397 +0000 UTC m=+2000.893894400" watchObservedRunningTime="2026-11-01 22:56:59.577190626 +0000 UTC m=+2000.894840628"
Nov 01 22:56:59.913462 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:56:59.913425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:00.337014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:00.332334    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:01.497287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:01.497254    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:01.974972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:01.974928    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"]
Nov 01 22:57:01.980643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:01.980620    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.090756 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.033790    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-71505e-openshift-default-dockercfg-xljd5\""
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.121646    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.121768    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.121836    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.121864    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.121908    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.122011    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.122105    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.122154    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfcmq\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.122297    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.137614 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.128859    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"]
Nov 01 22:57:02.178489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.178445    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:02.233068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.225675    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.233611 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.233478    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.234187 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.234039    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.234740 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.234574    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.235262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.235147    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.257391 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.257361    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.257911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.257887    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.258089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.258073    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfcmq\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.258346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.258331    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.258948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.258926    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.260130 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.260109    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.261770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.261183    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.265099 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.262029    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.266300 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.262251    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.278700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.278671    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"]
Nov 01 22:57:02.279137 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.279107    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy" containerID="cri-o://8b27f3a2b125d09b987a39fd8c8790789fdb7da4adbf391b4c84d3ccce56bee9" gracePeriod=30
Nov 01 22:57:02.319330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.318862    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"]
Nov 01 22:57:02.319330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.319122    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" containerID="cri-o://a2f04b780171f59c0212cfb22ac4e57549673b601e5fdd1d5828a06868616710" gracePeriod=30
Nov 01 22:57:02.320459 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.320399    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.320883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.320858    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.323081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.323035    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.327199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.327177    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfcmq\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq\") pod \"e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") " pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:02.332824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.332795    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"]
Nov 01 22:57:02.333394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.333325    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy" containerID="cri-o://9b5328ecc375649949c031911b11fbf98606f5abf9fa50072fa01633b51ff63f" gracePeriod=30
Nov 01 22:57:02.338484 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.338461    4531 patch_prober.go:29] interesting pod/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:57:02.338590 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.338512    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:57:02.592093 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.591996    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:02.598384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:02.598351    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:03.049324 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.049295    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"]
Nov 01 22:57:03.050986 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:57:03.050939    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c400087_2db5_4bbd_9427_be06e9acbf8f.slice/crio-13296f788da8978e2c6a7b06e44c9f8409db3614b10c63772a656289784b1589 WatchSource:0}: Error finding container 13296f788da8978e2c6a7b06e44c9f8409db3614b10c63772a656289784b1589: Status 404 returned error can't find the container with id 13296f788da8978e2c6a7b06e44c9f8409db3614b10c63772a656289784b1589
Nov 01 22:57:03.054409 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.054366    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:57:03.054522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.054477    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:57:03.054599 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.054527    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:57:03.436178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.436085    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" event={"ID":"6c400087-2db5-4bbd-9427-be06e9acbf8f","Type":"ContainerStarted","Data":"11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"}
Nov 01 22:57:03.436178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.436162    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" event={"ID":"6c400087-2db5-4bbd-9427-be06e9acbf8f","Type":"ContainerStarted","Data":"13296f788da8978e2c6a7b06e44c9f8409db3614b10c63772a656289784b1589"}
Nov 01 22:57:03.552896 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.552838    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podStartSLOduration=2.5528188480000003 podStartE2EDuration="2.552818848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:03.551133987 +0000 UTC m=+2004.868784002" watchObservedRunningTime="2026-11-01 22:57:03.552818848 +0000 UTC m=+2004.870468852"
Nov 01 22:57:03.599185 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:03.599157    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:04.605531 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:04.605499    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:04.919561 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:04.919486    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"]
Nov 01 22:57:04.924075 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:04.924056    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:04.975749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:04.975698    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-103369-serving-cert\""
Nov 01 22:57:04.993499 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:04.992176    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"]
Nov 01 22:57:05.017911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.017875    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhwc2\" (UniqueName: \"kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.018047 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.017931    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.018127 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.018042    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.018192 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.018146    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.067132 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.067107    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:05.119563 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.119533    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.119893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.119871    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhwc2\" (UniqueName: \"kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.119955 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.119938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.120015 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.119987    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.120765 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.120716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.122629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.122589    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.122842 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.122824    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.174039 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.173943    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhwc2\" (UniqueName: \"kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2\") pod \"maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.239107 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.239061    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:05.286296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.286263    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:05.453564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.453535    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:05.455145 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.455122    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:57:05.481941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.481914    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:05.697579 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.697551    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"]
Nov 01 22:57:05.698228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:05.698204    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 22:57:06.464532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:06.464494    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk" event={"ID":"0a65c30d-fd4d-4540-a8fc-f7e83fa24142","Type":"ContainerStarted","Data":"ee5370b2aa2bb1cb4f43b9f5e45823caaca0bda920dea01d43152600b6246a94"}
Nov 01 22:57:07.485815 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.485779    4531 generic.go:356] "Generic (PLEG): container finished" podID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerID="9b5328ecc375649949c031911b11fbf98606f5abf9fa50072fa01633b51ff63f" exitCode=0
Nov 01 22:57:07.486248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.485968    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" event={"ID":"937a1115-7ed7-4a43-bc36-1c2424f7228a","Type":"ContainerDied","Data":"9b5328ecc375649949c031911b11fbf98606f5abf9fa50072fa01633b51ff63f"}
Nov 01 22:57:07.488696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.488508    4531 generic.go:356] "Generic (PLEG): container finished" podID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerID="a2f04b780171f59c0212cfb22ac4e57549673b601e5fdd1d5828a06868616710" exitCode=0
Nov 01 22:57:07.488696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.488604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" event={"ID":"fd61e736-df0b-453a-a0c5-afbf11ac90a4","Type":"ContainerDied","Data":"a2f04b780171f59c0212cfb22ac4e57549673b601e5fdd1d5828a06868616710"}
Nov 01 22:57:07.495393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.490906    4531 generic.go:356] "Generic (PLEG): container finished" podID="113370e8-9731-4fb5-b6b9-474cc477c413" containerID="8b27f3a2b125d09b987a39fd8c8790789fdb7da4adbf391b4c84d3ccce56bee9" exitCode=0
Nov 01 22:57:07.495393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.491008    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" event={"ID":"113370e8-9731-4fb5-b6b9-474cc477c413","Type":"ContainerDied","Data":"8b27f3a2b125d09b987a39fd8c8790789fdb7da4adbf391b4c84d3ccce56bee9"}
Nov 01 22:57:07.495393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.495215    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk" event={"ID":"0a65c30d-fd4d-4540-a8fc-f7e83fa24142","Type":"ContainerStarted","Data":"16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9"}
Nov 01 22:57:07.495393 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.495284    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:07.514195 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.514120    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:07.644919 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.644792    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk" podStartSLOduration=2.437832166 podStartE2EDuration="3.644771348s" totalImagesPullingTime="1.206939182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:07.632438269 +0000 UTC m=+2008.950088271" watchObservedRunningTime="2026-11-01 22:57:07.644771348 +0000 UTC m=+2008.962421350"
Nov 01 22:57:07.706086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.703395    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:57:07.734014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.733985    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:57:07.753766 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.751922    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:07.754490 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.754466    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:57:07.757243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.757217    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.757447 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.757428    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.757954 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.757933    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.758410 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.758393    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.758846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.758828    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.759288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.759271    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.759655 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.759638    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.783010 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.782981    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.791849 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.791821    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.792104 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792087    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.792244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792228    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.792379 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792363    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.792509 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792497    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.792609 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792597    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.792717 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792704    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.792845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.792831    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.804501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.804474    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.815753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.808673    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.816053 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.816031    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.854235 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.854199    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.854517 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.854494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.854661 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.854650    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.854893 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.854861    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy\") pod \"113370e8-9731-4fb5-b6b9-474cc477c413\" (UID: \"113370e8-9731-4fb5-b6b9-474cc477c413\") "
Nov 01 22:57:07.855011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.854931    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.855011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.854973    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.757578    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.758334    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.758705    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.759169    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.759573    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.785865    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.785969    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.788094    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.796500    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.800984    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.803867    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.804015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "kube-api-access-wwclx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.804066    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.804109    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.804391    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.808549    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.816486    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.817248    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "kube-api-access-qk7h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.858087    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.860970    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.867894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.867822    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:57:07.891823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.891722    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:57:07.911774 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.911028    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.915747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.913639    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "kube-api-access-q8f7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:57:07.919134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.918422    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo" pod "113370e8-9731-4fb5-b6b9-474cc477c413" (UID: "113370e8-9731-4fb5-b6b9-474cc477c413"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:57:07.956604 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.956571    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy\") pod \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\" (UID: \"fd61e736-df0b-453a-a0c5-afbf11ac90a4\") "
Nov 01 22:57:07.956810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.956611    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket\") pod \"937a1115-7ed7-4a43-bc36-1c2424f7228a\" (UID: \"937a1115-7ed7-4a43-bc36-1c2424f7228a\") "
Nov 01 22:57:07.956946 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.956918    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket" pod "937a1115-7ed7-4a43-bc36-1c2424f7228a" (UID: "937a1115-7ed7-4a43-bc36-1c2424f7228a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:07.957097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957076    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8f7c\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-kube-api-access-q8f7c\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957098    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957108    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957117    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957126    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957134    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957143    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957153    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957162    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957170    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957179    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957193    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957204    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957219    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957228    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957232 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957237    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957245    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwclx\" (UniqueName: \"kubernetes.io/projected/937a1115-7ed7-4a43-bc36-1c2424f7228a-kube-api-access-wwclx\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957254    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/937a1115-7ed7-4a43-bc36-1c2424f7228a-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957262    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957270    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113370e8-9731-4fb5-b6b9-474cc477c413-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957278    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957286    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957295    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk7h9\" (UniqueName: \"kubernetes.io/projected/113370e8-9731-4fb5-b6b9-474cc477c413-kube-api-access-qk7h9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957303    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/937a1115-7ed7-4a43-bc36-1c2424f7228a-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957311    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113370e8-9731-4fb5-b6b9-474cc477c413-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.957644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.957318    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113370e8-9731-4fb5-b6b9-474cc477c413-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:07.958797 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:07.958774    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy" pod "fd61e736-df0b-453a-a0c5-afbf11ac90a4" (UID: "fd61e736-df0b-453a-a0c5-afbf11ac90a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:57:08.059108 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.059075    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd61e736-df0b-453a-a0c5-afbf11ac90a4-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:57:08.506166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.506129    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"
Nov 01 22:57:08.506166 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.506149    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l" event={"ID":"937a1115-7ed7-4a43-bc36-1c2424f7228a","Type":"ContainerDied","Data":"472c0d610ee8aac98fd57e16001157999e92bf9b4e78943ce79139ea71e237f6"}
Nov 01 22:57:08.506753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.506218    4531 scope.go:122] "RemoveContainer" containerID="9b5328ecc375649949c031911b11fbf98606f5abf9fa50072fa01633b51ff63f"
Nov 01 22:57:08.508289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.508263    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"
Nov 01 22:57:08.508289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.508271    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26" event={"ID":"fd61e736-df0b-453a-a0c5-afbf11ac90a4","Type":"ContainerDied","Data":"d57a1dcdffbbc0ec1ec7981e28bf5464df3110fc66c4bf7d4ca16215351f3317"}
Nov 01 22:57:08.510814 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.510783    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn" event={"ID":"113370e8-9731-4fb5-b6b9-474cc477c413","Type":"ContainerDied","Data":"11a803db9d250fd8bb0b86636b1a084b85915abcab8e89a6be4d486c7beadd9e"}
Nov 01 22:57:08.510947 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.510880    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"
Nov 01 22:57:08.522512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.522412    4531 scope.go:122] "RemoveContainer" containerID="a2f04b780171f59c0212cfb22ac4e57549673b601e5fdd1d5828a06868616710"
Nov 01 22:57:08.533321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.533300    4531 scope.go:122] "RemoveContainer" containerID="8b27f3a2b125d09b987a39fd8c8790789fdb7da4adbf391b4c84d3ccce56bee9"
Nov 01 22:57:08.675911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.675827    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"]
Nov 01 22:57:08.682753 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.682700    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-t28jn"]
Nov 01 22:57:08.780377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.780338    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"]
Nov 01 22:57:08.789999 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.789953    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-55x5l"]
Nov 01 22:57:08.818532 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.818504    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" path="/var/lib/kubelet/pods/113370e8-9731-4fb5-b6b9-474cc477c413/volumes"
Nov 01 22:57:08.819068 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.819049    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" path="/var/lib/kubelet/pods/937a1115-7ed7-4a43-bc36-1c2424f7228a/volumes"
Nov 01 22:57:08.893144 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.893108    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"]
Nov 01 22:57:08.903677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:08.903644    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-bms26"]
Nov 01 22:57:10.381272 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:10.381243    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 22:57:10.381822 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:10.381323    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 22:57:10.632790 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:10.631452    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:10.818200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:10.818164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:10.819744 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:10.819707    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" path="/var/lib/kubelet/pods/fd61e736-df0b-453a-a0c5-afbf11ac90a4/volumes"
Nov 01 22:57:11.011014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.010980    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:11.281303 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.281228    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"]
Nov 01 22:57:11.281970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.281955    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy"
Nov 01 22:57:11.281970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.281971    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy"
Nov 01 22:57:11.282089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.281981    4531 cpu_manager.go:414] "containerMap: removing container" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy"
Nov 01 22:57:11.282089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.281986    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy"
Nov 01 22:57:11.282089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.282018    4531 cpu_manager.go:414] "containerMap: removing container" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy"
Nov 01 22:57:11.282089 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.282022    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy"
Nov 01 22:57:11.282219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.282127    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd61e736-df0b-453a-a0c5-afbf11ac90a4" containerName="istio-proxy"
Nov 01 22:57:11.282219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.282136    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="937a1115-7ed7-4a43-bc36-1c2424f7228a" containerName="istio-proxy"
Nov 01 22:57:11.282219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.282148    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="113370e8-9731-4fb5-b6b9-474cc477c413" containerName="istio-proxy"
Nov 01 22:57:11.287836 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.287812    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.339013 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.338976    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-71505e\""
Nov 01 22:57:11.339873 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.339584    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-71505e-dockercfg-f4xxv\""
Nov 01 22:57:11.343681 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.343655    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"]
Nov 01 22:57:11.419016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.418975    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.419503 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.419039    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vww6q\" (UniqueName: \"kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q\") pod \"payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.491243 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.491049    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"]
Nov 01 22:57:11.517748 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.514915    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.520757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.520571    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.520757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.520615    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vww6q\" (UniqueName: \"kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q\") pod \"payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.520757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.520653    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgqsg\" (UniqueName: \"kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg\") pod \"payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.521059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.520778    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.522080 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.522016    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume\") pod \"payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.545823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.545434    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"]
Nov 01 22:57:11.575200 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.575165    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vww6q\" (UniqueName: \"kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q\") pod \"payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") " pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.613331 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.607971    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:11.624959 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.624928    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.625106 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.625085    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgqsg\" (UniqueName: \"kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg\") pod \"payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.626011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.625990    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.679325 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.679297    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgqsg\" (UniqueName: \"kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg\") pod \"payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:11.877234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:11.877097    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:12.002191 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.002157    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"]
Nov 01 22:57:12.004381 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:57:12.004320    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b8ec40_75be_4e2d_b263_f69b487c41ec.slice/crio-11f215fa5bf1645a4ffcb5487dfd3264598f769290e4d502fb59aae3808088f3 WatchSource:0}: Error finding container 11f215fa5bf1645a4ffcb5487dfd3264598f769290e4d502fb59aae3808088f3: Status 404 returned error can't find the container with id 11f215fa5bf1645a4ffcb5487dfd3264598f769290e4d502fb59aae3808088f3
Nov 01 22:57:12.274899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.274869    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"]
Nov 01 22:57:12.278469 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:57:12.278440    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf3a948_317a_47d9_aef5_81303c4f5400.slice/crio-9945753408fa04d0d798dc03533276e5c7456eb593af5b80d2afac47231ceb61 WatchSource:0}: Error finding container 9945753408fa04d0d798dc03533276e5c7456eb593af5b80d2afac47231ceb61: Status 404 returned error can't find the container with id 9945753408fa04d0d798dc03533276e5c7456eb593af5b80d2afac47231ceb61
Nov 01 22:57:12.527110 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:57:12.527024    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:57:12.661700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.661647    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr" event={"ID":"67b8ec40-75be-4e2d-b263-f69b487c41ec","Type":"ContainerStarted","Data":"d7abf28934cd61f8b5a66458ca562e8466fe4850c81c708202352f2b7bd89641"}
Nov 01 22:57:12.661890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.661783    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:12.661890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.661812    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr" event={"ID":"67b8ec40-75be-4e2d-b263-f69b487c41ec","Type":"ContainerStarted","Data":"11f215fa5bf1645a4ffcb5487dfd3264598f769290e4d502fb59aae3808088f3"}
Nov 01 22:57:12.663298 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.663273    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l" event={"ID":"fbf3a948-317a-47d9-aef5-81303c4f5400","Type":"ContainerStarted","Data":"303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89"}
Nov 01 22:57:12.663405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.663313    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l" event={"ID":"fbf3a948-317a-47d9-aef5-81303c4f5400","Type":"ContainerStarted","Data":"9945753408fa04d0d798dc03533276e5c7456eb593af5b80d2afac47231ceb61"}
Nov 01 22:57:12.663405 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.663363    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:12.771806 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.771757    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr" podStartSLOduration=1.7717454780000002 podStartE2EDuration="1.771745478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:12.76888072 +0000 UTC m=+2014.086530722" watchObservedRunningTime="2026-11-01 22:57:12.771745478 +0000 UTC m=+2014.089395473"
Nov 01 22:57:12.878679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:12.878575    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l" podStartSLOduration=1.878559423 podStartE2EDuration="1.878559423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:12.875842119 +0000 UTC m=+2014.193492146" watchObservedRunningTime="2026-11-01 22:57:12.878559423 +0000 UTC m=+2014.196209425"
Nov 01 22:57:14.529486 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:14.529459    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 22:57:15.630694 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:15.630646    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:20.360693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:20.360612    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:23.674267 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:23.674236    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 22:57:23.674693 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:23.674316    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 22:57:24.725798 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.725766    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"]
Nov 01 22:57:24.730560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.730541    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.779813 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.779786    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-945efd02-serving-cert\""
Nov 01 22:57:24.786141 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.786120    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"]
Nov 01 22:57:24.838752 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.838709    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.838891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.838875    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjtt\" (UniqueName: \"kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.839001 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.838983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.839186 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.839165    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.940650 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.940617    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjtt\" (UniqueName: \"kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.940807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.940708    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.941182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.940909    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.941182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.940973    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.941182 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:57:24.940914    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-serving-cert: secret "maas-api-e2e-mt-945efd02-serving-cert" not found
Nov 01 22:57:24.941182 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:57:24.941068    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls podName:e7144b91-f7a8-4c94-8582-9f735c01b8a0 nodeName:}" failed. No retries permitted until 2026-11-01 22:57:25.4410361 +0000 UTC m=+2026.758686078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls") pod "maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" (UID: "e7144b91-f7a8-4c94-8582-9f735c01b8a0") : secret "maas-api-e2e-mt-945efd02-serving-cert" not found
Nov 01 22:57:24.941620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.941596    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.943440 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.943416    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:24.997284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:24.997212    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjtt\" (UniqueName: \"kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:25.004103 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:25.004054    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:25.187697 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:25.187664    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:25.448470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:25.448435    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:25.451005 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:25.450978    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\") pod \"maas-api-e2e-mt-945efd02-79b65bbb9-6fq22\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:25.646400 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:25.646375    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:26.027223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:26.027189    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"]
Nov 01 22:57:26.029875 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:57:26.029838    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7144b91_f7a8_4c94_8582_9f735c01b8a0.slice/crio-86cf5f05423893bb5ca4c64a848781b50dec9e949d5ccb9cc12e0dc19f24050c WatchSource:0}: Error finding container 86cf5f05423893bb5ca4c64a848781b50dec9e949d5ccb9cc12e0dc19f24050c: Status 404 returned error can't find the container with id 86cf5f05423893bb5ca4c64a848781b50dec9e949d5ccb9cc12e0dc19f24050c
Nov 01 22:57:26.826458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:26.826418    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" event={"ID":"e7144b91-f7a8-4c94-8582-9f735c01b8a0","Type":"ContainerStarted","Data":"86cf5f05423893bb5ca4c64a848781b50dec9e949d5ccb9cc12e0dc19f24050c"}
Nov 01 22:57:27.824699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:27.824657    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" event={"ID":"e7144b91-f7a8-4c94-8582-9f735c01b8a0","Type":"ContainerStarted","Data":"77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea"}
Nov 01 22:57:27.825231 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:27.824750    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:27.939016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:27.938951    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" podStartSLOduration=3.073145558 podStartE2EDuration="3.938936864s" totalImagesPullingTime="865.791306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:27.935036518 +0000 UTC m=+2029.252686518" watchObservedRunningTime="2026-11-01 22:57:27.938936864 +0000 UTC m=+2029.256586912"
Nov 01 22:57:29.961634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:29.961602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:33.842360 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:33.842330    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:57:34.809438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:34.809310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:39.149278 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:39.149203    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:39.513091 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:39.513060    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:42.531597 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:57:42.531561    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:57:44.355515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:44.355481    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:49.051580 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:49.051550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:51.072605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.072565    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:51.740138 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.740104    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"]
Nov 01 22:57:51.747144 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.747118    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796288 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796253    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796296    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796426    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796496 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796487    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796582    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqb8n\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796616    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796646    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796680    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.796894 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.796849    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.819337 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.819309    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7da9ed-openshift-default-dockercfg-xrsgs\""
Nov 01 22:57:51.848613 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.848580    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"]
Nov 01 22:57:51.897888 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.897849    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.897962    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898030    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqb8n\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898084 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898076    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898125    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898170    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898266 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898233    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898314    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.898425 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.898345    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.899081 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.899053    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.900155 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.900134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.913778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.901038    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.913778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.901482    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.913778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.901686    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.913778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.901820    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.913778 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.912751    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.954514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.954483    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:51.954737 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:51.954697    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqb8n\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n\") pod \"e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") " pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:52.062833 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:52.062743    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:52.515972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:52.515939    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"]
Nov 01 22:57:52.521406 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:52.521372    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:57:52.521522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:52.521451    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:57:52.521522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:52.521496    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:57:53.058006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.056823    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" event={"ID":"290646e6-1e8b-41ff-9e0b-8d21674fbf9c","Type":"ContainerStarted","Data":"4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"}
Nov 01 22:57:53.058006 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.056899    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" event={"ID":"290646e6-1e8b-41ff-9e0b-8d21674fbf9c","Type":"ContainerStarted","Data":"6bd8c8c5508cedeced5181086eb34b53d46062d247648f4cb2e66fc2c2ac41d5"}
Nov 01 22:57:53.063711 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.063682    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:53.069924 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.069902    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:53.194294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.194241    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podStartSLOduration=2.194226928 podStartE2EDuration="2.194226928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:57:53.191716232 +0000 UTC m=+2054.509366233" watchObservedRunningTime="2026-11-01 22:57:53.194226928 +0000 UTC m=+2054.511876929"
Nov 01 22:57:53.740864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.740835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:53.956749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:53.955805    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:54.069428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:54.069344    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:54.070494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:54.070470    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 22:57:56.005927 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:56.005892    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:56.220887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:56.220856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:58.981161 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:58.981119    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:59.224865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.224835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:59.439887 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.439795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:57:59.768751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.764819    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"]
Nov 01 22:57:59.774072 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.774046    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:57:59.819703 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.819669    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"]
Nov 01 22:57:59.823800 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.823770    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7da9ed\""
Nov 01 22:57:59.860171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.860142    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-7da9ed-dockercfg-nwqbc\""
Nov 01 22:57:59.916751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.915742    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv8r\" (UniqueName: \"kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r\") pod \"payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:57:59.916751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.916097    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:57:59.931637 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.931601    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"]
Nov 01 22:57:59.946284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.946228    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:57:59.989253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:57:59.989220    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"]
Nov 01 22:58:00.017590 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.017552    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.017792 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.017612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:00.017892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.017798    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq5h2\" (UniqueName: \"kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2\") pod \"payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.017970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.017947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv8r\" (UniqueName: \"kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r\") pod \"payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:00.018465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.018445    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:00.072516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.072464    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv8r\" (UniqueName: \"kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r\") pod \"payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") " pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:00.093289 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.093269    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:00.119510 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.119488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.119628 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.119581    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq5h2\" (UniqueName: \"kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2\") pod \"payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.120411 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.120388    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.174162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.174134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq5h2\" (UniqueName: \"kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2\") pod \"payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.266508 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.266477    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:00.469827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.469798    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"]
Nov 01 22:58:00.473888 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:00.473858    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49c48b03_dd06_4755_a8d6_f4475978e0cd.slice/crio-d351ae3b34c07f7cfe0b7bafbed45a1d5b5fad4c3e4ef42566ca66338fadd884 WatchSource:0}: Error finding container d351ae3b34c07f7cfe0b7bafbed45a1d5b5fad4c3e4ef42566ca66338fadd884: Status 404 returned error can't find the container with id d351ae3b34c07f7cfe0b7bafbed45a1d5b5fad4c3e4ef42566ca66338fadd884
Nov 01 22:58:00.646263 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:00.646239    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"]
Nov 01 22:58:00.649221 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:00.649196    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acc5535_72ee_4a61_87f1_ac8a28bc836b.slice/crio-a3cee84529900a2db301e13b49b214e8d78a3f6dfc240a5d283b78393ae83a7a WatchSource:0}: Error finding container a3cee84529900a2db301e13b49b214e8d78a3f6dfc240a5d283b78393ae83a7a: Status 404 returned error can't find the container with id a3cee84529900a2db301e13b49b214e8d78a3f6dfc240a5d283b78393ae83a7a
Nov 01 22:58:01.131991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.131955    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5" event={"ID":"49c48b03-dd06-4755-a8d6-f4475978e0cd","Type":"ContainerStarted","Data":"a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d"}
Nov 01 22:58:01.132460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.132008    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5" event={"ID":"49c48b03-dd06-4755-a8d6-f4475978e0cd","Type":"ContainerStarted","Data":"d351ae3b34c07f7cfe0b7bafbed45a1d5b5fad4c3e4ef42566ca66338fadd884"}
Nov 01 22:58:01.132460 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.132298    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:01.133764 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.133742    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x" event={"ID":"0acc5535-72ee-4a61-87f1-ac8a28bc836b","Type":"ContainerStarted","Data":"86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb"}
Nov 01 22:58:01.133905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.133777    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x" event={"ID":"0acc5535-72ee-4a61-87f1-ac8a28bc836b","Type":"ContainerStarted","Data":"a3cee84529900a2db301e13b49b214e8d78a3f6dfc240a5d283b78393ae83a7a"}
Nov 01 22:58:01.133905 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.133810    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:01.240416 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.240369    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5" podStartSLOduration=2.240356756 podStartE2EDuration="2.240356756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:01.238144192 +0000 UTC m=+2062.555794195" watchObservedRunningTime="2026-11-01 22:58:01.240356756 +0000 UTC m=+2062.558006757"
Nov 01 22:58:01.348348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:01.348297    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x" podStartSLOduration=2.348285246 podStartE2EDuration="2.348285246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:01.345951907 +0000 UTC m=+2062.663601910" watchObservedRunningTime="2026-11-01 22:58:01.348285246 +0000 UTC m=+2062.665935259"
Nov 01 22:58:04.219555 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:04.219516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:05.966647 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:05.964258    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"]
Nov 01 22:58:05.974079 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:05.974045    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.022120 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.022091    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"]
Nov 01 22:58:06.063162 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.063108    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:58:06.063300 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.063196    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:58:06.107041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.107006    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.107189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.107169    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.107321 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.107306    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pw2d\" (UniqueName: \"kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.208654 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.208626    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pw2d\" (UniqueName: \"kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.208850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.208719    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.208850 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.208842    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.209429 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.209406    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.211414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.211389    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.265002 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.264942    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pw2d\" (UniqueName: \"kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d\") pod \"maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.293899 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.293873    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:06.672970 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:06.672858    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"]
Nov 01 22:58:06.676685 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:06.676652    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538dbe35_3802_4360_857d_4948e966ed90.slice/crio-79ef2bbfacbcebc089574830ebf97d3d8b7f0376270b77f05ba5bc0e33c9dfd4 WatchSource:0}: Error finding container 79ef2bbfacbcebc089574830ebf97d3d8b7f0376270b77f05ba5bc0e33c9dfd4: Status 404 returned error can't find the container with id 79ef2bbfacbcebc089574830ebf97d3d8b7f0376270b77f05ba5bc0e33c9dfd4
Nov 01 22:58:07.188649 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:07.188604    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k" event={"ID":"538dbe35-3802-4360-857d-4948e966ed90","Type":"ContainerStarted","Data":"3c93aed735564957213b41a5c62a90ee372d42e84b45dfeec5c6dbaac8a30f84"}
Nov 01 22:58:07.189167 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:07.188672    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k" event={"ID":"538dbe35-3802-4360-857d-4948e966ed90","Type":"ContainerStarted","Data":"79ef2bbfacbcebc089574830ebf97d3d8b7f0376270b77f05ba5bc0e33c9dfd4"}
Nov 01 22:58:07.300276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:07.300168    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k" podStartSLOduration=2.300150701 podStartE2EDuration="2.300150701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:07.29555493 +0000 UTC m=+2068.613204932" watchObservedRunningTime="2026-11-01 22:58:07.300150701 +0000 UTC m=+2068.617800701"
Nov 01 22:58:08.142677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.142644    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"]
Nov 01 22:58:08.147432 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.147410    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.196057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.196019    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-945efd02-openshift-default-dockercfg-fl45l\""
Nov 01 22:58:08.198247 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.198220    4531 generic.go:356] "Generic (PLEG): container finished" podID="538dbe35-3802-4360-857d-4948e966ed90" containerID="3c93aed735564957213b41a5c62a90ee372d42e84b45dfeec5c6dbaac8a30f84" exitCode=0
Nov 01 22:58:08.198382 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.198258    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k" event={"ID":"538dbe35-3802-4360-857d-4948e966ed90","Type":"ContainerDied","Data":"3c93aed735564957213b41a5c62a90ee372d42e84b45dfeec5c6dbaac8a30f84"}
Nov 01 22:58:08.234772 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.234714    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.234890 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.234784    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.234960 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.234931    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.235184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.235033    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.235184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.235069    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.235184 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.235133    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd49q\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.235371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.235199    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.235371 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.235321    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.235458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.235435    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.248582 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.248558    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"]
Nov 01 22:58:08.336932 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.336899    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.336956    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337007    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd49q\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337113 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337063    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337173    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337258    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337296    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337317    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337402    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337544 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337446    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337677    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337810 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337696    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.337885 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.337863    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.338036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.338014    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.339556 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.339528    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.339664 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.339629    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.391323 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.391281    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.394093 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.394015    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd49q\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.463270 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.463242    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:08.895839 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.895807    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"]
Nov 01 22:58:08.899935 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:08.899906    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3e3eed_c404_4205_b21f_fc5d8c982229.slice/crio-c8a59d4309895caa92faebde4efa7fa3ddec216c074b73e7605cea014d94b62a WatchSource:0}: Error finding container c8a59d4309895caa92faebde4efa7fa3ddec216c074b73e7605cea014d94b62a: Status 404 returned error can't find the container with id c8a59d4309895caa92faebde4efa7fa3ddec216c074b73e7605cea014d94b62a
Nov 01 22:58:08.903696 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.903661    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:58:08.903845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.903755    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:58:08.903845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:08.903807    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:58:09.208443 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.208409    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" event={"ID":"ce3e3eed-c404-4205-b21f-fc5d8c982229","Type":"ContainerStarted","Data":"7fac0d27fcfa5dc9979c6deae183ccc07811135a5bc373fd01a849eacd96f7e2"}
Nov 01 22:58:09.208913 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.208475    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" event={"ID":"ce3e3eed-c404-4205-b21f-fc5d8c982229","Type":"ContainerStarted","Data":"c8a59d4309895caa92faebde4efa7fa3ddec216c074b73e7605cea014d94b62a"}
Nov 01 22:58:09.404180 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.404150    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:09.457718 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.457690    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d\" (UniqueName: \"kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d\") pod \"538dbe35-3802-4360-857d-4948e966ed90\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") "
Nov 01 22:58:09.457881 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.457756    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca\") pod \"538dbe35-3802-4360-857d-4948e966ed90\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") "
Nov 01 22:58:09.458031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.458009    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca\") pod \"538dbe35-3802-4360-857d-4948e966ed90\" (UID: \"538dbe35-3802-4360-857d-4948e966ed90\") "
Nov 01 22:58:09.458438 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.458414    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca" pod "538dbe35-3802-4360-857d-4948e966ed90" (UID: "538dbe35-3802-4360-857d-4948e966ed90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:09.459339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.459278    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/538dbe35-3802-4360-857d-4948e966ed90-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:09.460154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.460131    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d" pod "538dbe35-3802-4360-857d-4948e966ed90" (UID: "538dbe35-3802-4360-857d-4948e966ed90"). InnerVolumeSpecName "kube-api-access-6pw2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:09.460261 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.460220    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca" pod "538dbe35-3802-4360-857d-4948e966ed90" (UID: "538dbe35-3802-4360-857d-4948e966ed90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:58:09.460953 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.460931    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:09.464287 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.464270    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:09.465216 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.465196    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:09.465284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.465238    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:09.517703 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.517636    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podStartSLOduration=1.517616122 podStartE2EDuration="1.517616122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:09.323171412 +0000 UTC m=+2070.640821413" watchObservedRunningTime="2026-11-01 22:58:09.517616122 +0000 UTC m=+2070.835266126"
Nov 01 22:58:09.560522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.560489    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pw2d\" (UniqueName: \"kubernetes.io/projected/538dbe35-3802-4360-857d-4948e966ed90-kube-api-access-6pw2d\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:09.560522 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:09.560521    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/538dbe35-3802-4360-857d-4948e966ed90-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:10.225721 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:10.225691    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"
Nov 01 22:58:10.226163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:10.225720    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k" event={"ID":"538dbe35-3802-4360-857d-4948e966ed90","Type":"ContainerDied","Data":"79ef2bbfacbcebc089574830ebf97d3d8b7f0376270b77f05ba5bc0e33c9dfd4"}
Nov 01 22:58:10.226163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:10.225803    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79ef2bbfacbcebc089574830ebf97d3d8b7f0376270b77f05ba5bc0e33c9dfd4"
Nov 01 22:58:10.484805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:10.484196    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:10.484805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:10.484258    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:11.464133 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:11.464097    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:11.485920 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:11.469791    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:12.145916 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.145886    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 22:58:12.146098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.145960    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 22:58:12.436140 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.435961    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"]
Nov 01 22:58:12.436750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.436693    4531 cpu_manager.go:414] "containerMap: removing container" podUID="538dbe35-3802-4360-857d-4948e966ed90" containerName="revoke-keys"
Nov 01 22:58:12.436750 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.436709    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="538dbe35-3802-4360-857d-4948e966ed90" containerName="revoke-keys"
Nov 01 22:58:12.436957 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.436891    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="538dbe35-3802-4360-857d-4948e966ed90" containerName="revoke-keys"
Nov 01 22:58:12.440990 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.440971    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.510749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.506654    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:58:12.510749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.506742    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:58:12.530457 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.530410    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.530639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.530497    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stns7\" (UniqueName: \"kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.530639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.530570    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:12.530639 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.530616    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:12.530865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.530663    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.530865 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.530829    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"]
Nov 01 22:58:12.609349 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:58:12.609308    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:58:12.632605 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.632579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stns7\" (UniqueName: \"kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.632948 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.632927    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.633423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.633399    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.633601 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.633574    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.639326 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.639305    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.699805 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.699446    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stns7\" (UniqueName: \"kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7\") pod \"maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:12.755699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:12.755669    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:13.198803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:13.198413    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"]
Nov 01 22:58:13.204323 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:13.204297    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9987bf72_4ac2_471f_989b_6ec9b2e07376.slice/crio-4fa7c76f7d1a67159e58e5e747ac68916fc1adb7888037be0301af92cdd17132 WatchSource:0}: Error finding container 4fa7c76f7d1a67159e58e5e747ac68916fc1adb7888037be0301af92cdd17132: Status 404 returned error can't find the container with id 4fa7c76f7d1a67159e58e5e747ac68916fc1adb7888037be0301af92cdd17132
Nov 01 22:58:13.252612 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:13.252582    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl" event={"ID":"9987bf72-4ac2-471f-989b-6ec9b2e07376","Type":"ContainerStarted","Data":"4fa7c76f7d1a67159e58e5e747ac68916fc1adb7888037be0301af92cdd17132"}
Nov 01 22:58:13.478824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:13.473818    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:13.478824 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:13.473878    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:14.298757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:14.298292    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl" event={"ID":"9987bf72-4ac2-471f-989b-6ec9b2e07376","Type":"ContainerStarted","Data":"8e76e9735f11afffe58bef59e7a8c05a3d54279aa65b91863f844d43c627e040"}
Nov 01 22:58:14.405807 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:14.405715    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl" podStartSLOduration=2.405699576 podStartE2EDuration="2.405699576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:14.405395252 +0000 UTC m=+2075.723045274" watchObservedRunningTime="2026-11-01 22:58:14.405699576 +0000 UTC m=+2075.723349578"
Nov 01 22:58:14.464470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:14.464044    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:14.464470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:14.464109    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:15.040276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:15.040244    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:15.312540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:15.312449    4531 generic.go:356] "Generic (PLEG): container finished" podID="9987bf72-4ac2-471f-989b-6ec9b2e07376" containerID="8e76e9735f11afffe58bef59e7a8c05a3d54279aa65b91863f844d43c627e040" exitCode=0
Nov 01 22:58:15.312540 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:15.312486    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl" event={"ID":"9987bf72-4ac2-471f-989b-6ec9b2e07376","Type":"ContainerDied","Data":"8e76e9735f11afffe58bef59e7a8c05a3d54279aa65b91863f844d43c627e040"}
Nov 01 22:58:15.463979 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:15.463941    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused" start-of-body=
Nov 01 22:58:15.464168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:15.464006    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.184:15021/healthz/ready\": dial tcp 10.133.0.184:15021: connect: connection refused"
Nov 01 22:58:16.469014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.468988    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:16.469412 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.469373    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:16.470163 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.470143    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 22:58:16.491339 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.491323    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:16.628911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.628809    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7\" (UniqueName: \"kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7\") pod \"9987bf72-4ac2-471f-989b-6ec9b2e07376\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") "
Nov 01 22:58:16.629090 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.628967    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca\") pod \"9987bf72-4ac2-471f-989b-6ec9b2e07376\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") "
Nov 01 22:58:16.629154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.629087    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca\") pod \"9987bf72-4ac2-471f-989b-6ec9b2e07376\" (UID: \"9987bf72-4ac2-471f-989b-6ec9b2e07376\") "
Nov 01 22:58:16.629299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.629272    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca" pod "9987bf72-4ac2-471f-989b-6ec9b2e07376" (UID: "9987bf72-4ac2-471f-989b-6ec9b2e07376"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:16.629803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.629782    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:16.631056 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.631033    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca" pod "9987bf72-4ac2-471f-989b-6ec9b2e07376" (UID: "9987bf72-4ac2-471f-989b-6ec9b2e07376"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:58:16.631248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.631229    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7" pod "9987bf72-4ac2-471f-989b-6ec9b2e07376" (UID: "9987bf72-4ac2-471f-989b-6ec9b2e07376"). InnerVolumeSpecName "kube-api-access-stns7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:16.731241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.731208    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stns7\" (UniqueName: \"kubernetes.io/projected/9987bf72-4ac2-471f-989b-6ec9b2e07376-kube-api-access-stns7\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:16.731241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.731240    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9987bf72-4ac2-471f-989b-6ec9b2e07376-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:16.969976 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:16.969934    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:17.160606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:17.160565    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:17.330021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:17.329937    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"
Nov 01 22:58:17.330021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:17.329968    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl" event={"ID":"9987bf72-4ac2-471f-989b-6ec9b2e07376","Type":"ContainerDied","Data":"4fa7c76f7d1a67159e58e5e747ac68916fc1adb7888037be0301af92cdd17132"}
Nov 01 22:58:17.330249 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:17.330055    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fa7c76f7d1a67159e58e5e747ac68916fc1adb7888037be0301af92cdd17132"
Nov 01 22:58:20.260702 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:20.260661    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:22.501063 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:22.501024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:22.692242 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:22.692208    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:23.283199 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.283158    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"]
Nov 01 22:58:23.284059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.284029    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9987bf72-4ac2-471f-989b-6ec9b2e07376" containerName="revoke-keys"
Nov 01 22:58:23.284059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.284058    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9987bf72-4ac2-471f-989b-6ec9b2e07376" containerName="revoke-keys"
Nov 01 22:58:23.284276 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.284169    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9987bf72-4ac2-471f-989b-6ec9b2e07376" containerName="revoke-keys"
Nov 01 22:58:23.288588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.288567    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.398634 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.398597    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"]
Nov 01 22:58:23.436035 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436005    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436075    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436175    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436404    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436501 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436434    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436558 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436538    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436600 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436579    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436640 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.436679 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.436665    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824bh\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538254    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538346 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538304    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824bh\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538353    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538426    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538473    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538667    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538666    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538692    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538795    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.538883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538839    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.539376 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.538961    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.539376 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.539227    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.539376 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.539361    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.539560 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.539540    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.541422 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.541393    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.541616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.541599    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.592228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.592196    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.592363 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.592291    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-824bh\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh\") pod \"e2e-mt-945efd02-openshift-default-974cc8b4f-8w784\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") " pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:23.603700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:23.603681    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:24.051042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.050674    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"]
Nov 01 22:58:24.053624 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:24.053595    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06062fa0_7733_4503_b2b6_10065547bf34.slice/crio-d5ada9ec8369e65d50681f8c3294a7b9b7ebe4b13a6a6ef995635a133aa22e4c WatchSource:0}: Error finding container d5ada9ec8369e65d50681f8c3294a7b9b7ebe4b13a6a6ef995635a133aa22e4c: Status 404 returned error can't find the container with id d5ada9ec8369e65d50681f8c3294a7b9b7ebe4b13a6a6ef995635a133aa22e4c
Nov 01 22:58:24.056549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.056517    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:58:24.056624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.056581    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:58:24.056624 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.056610    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:58:24.372322 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.372226    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:24.436011 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.435975    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" event={"ID":"06062fa0-7733-4503-b2b6-10065547bf34","Type":"ContainerStarted","Data":"68a755e374a55ff1f0e97cc71244c52ea909a6b592f2483f70b41fce3d9f24ed"}
Nov 01 22:58:24.436320 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.436296    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" event={"ID":"06062fa0-7733-4503-b2b6-10065547bf34","Type":"ContainerStarted","Data":"d5ada9ec8369e65d50681f8c3294a7b9b7ebe4b13a6a6ef995635a133aa22e4c"}
Nov 01 22:58:24.559988 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.559870    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:24.565722 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.565668    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podStartSLOduration=1.5656529460000002 podStartE2EDuration="1.565652946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:58:24.559336065 +0000 UTC m=+2085.876986076" watchObservedRunningTime="2026-11-01 22:58:24.565652946 +0000 UTC m=+2085.883302946"
Nov 01 22:58:24.614747 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:24.613844    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:25.535042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:25.534911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:25.608234 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:25.608190    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:58:25.608831 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:25.608783    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 01 22:58:26.604257 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:26.604218    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:58:26.604463 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:26.604276    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 01 22:58:27.604799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:27.604760    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:58:27.605275 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:27.604824    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 01 22:58:28.604891 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:28.604851    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:58:28.605407 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:28.604932    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 01 22:58:29.604311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:29.604280    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:58:29.604509 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:29.604331    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 01 22:58:30.607219 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:30.607173    4531 patch_prober.go:29] interesting pod/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused" start-of-body=
Nov 01 22:58:30.607898 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:30.607375    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.186:15021/healthz/ready\": dial tcp 10.133.0.186:15021: connect: connection refused"
Nov 01 22:58:30.632311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:30.632273    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:31.610660 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.610632    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:31.611178 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.611156    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:31.611823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.611803    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 22:58:31.817993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.817956    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"]
Nov 01 22:58:31.822923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.822901    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:31.873160 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.873096    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"]
Nov 01 22:58:31.875269 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.875245    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:58:31.878311 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.876227    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:58:31.923171 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.923143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:31.923353 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.923186    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n6jt\" (UniqueName: \"kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:31.923430 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:31.923416    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.025376 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.025343    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.025554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.025510    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.025554 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.025530    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n6jt\" (UniqueName: \"kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.026179 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.026154    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.027712 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.027691    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.079588 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.079554    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n6jt\" (UniqueName: \"kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.138515 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.138457    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:32.529991 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:32.529961    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"]
Nov 01 22:58:32.533510 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:58:32.533477    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod548be2ce_a087_45ed_b703_d2e322966045.slice/crio-259396d630939776c1a37ae3450eecce64bc09b38dd28510feacd9f3d0ab49fc WatchSource:0}: Error finding container 259396d630939776c1a37ae3450eecce64bc09b38dd28510feacd9f3d0ab49fc: Status 404 returned error can't find the container with id 259396d630939776c1a37ae3450eecce64bc09b38dd28510feacd9f3d0ab49fc
Nov 01 22:58:33.546296 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:33.546271    4531 generic.go:356] "Generic (PLEG): container finished" podID="548be2ce-a087-45ed-b703-d2e322966045" containerID="45020eed86ce448f0f4363508c2026f159c390a832e619eefdf6cde8b04d9627" exitCode=0
Nov 01 22:58:33.546635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:33.546356    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg" event={"ID":"548be2ce-a087-45ed-b703-d2e322966045","Type":"ContainerDied","Data":"45020eed86ce448f0f4363508c2026f159c390a832e619eefdf6cde8b04d9627"}
Nov 01 22:58:33.546635 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:33.546408    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg" event={"ID":"548be2ce-a087-45ed-b703-d2e322966045","Type":"ContainerStarted","Data":"259396d630939776c1a37ae3450eecce64bc09b38dd28510feacd9f3d0ab49fc"}
Nov 01 22:58:34.343253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.343220    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:34.534821 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.534794    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:34.736512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.736490    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:34.862700 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.862656    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca\") pod \"548be2ce-a087-45ed-b703-d2e322966045\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") "
Nov 01 22:58:34.862902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.862826    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt\" (UniqueName: \"kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt\") pod \"548be2ce-a087-45ed-b703-d2e322966045\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") "
Nov 01 22:58:34.862902 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.862893    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca\") pod \"548be2ce-a087-45ed-b703-d2e322966045\" (UID: \"548be2ce-a087-45ed-b703-d2e322966045\") "
Nov 01 22:58:34.863644 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.863611    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca" pod "548be2ce-a087-45ed-b703-d2e322966045" (UID: "548be2ce-a087-45ed-b703-d2e322966045"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:58:34.865795 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.865772    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca" pod "548be2ce-a087-45ed-b703-d2e322966045" (UID: "548be2ce-a087-45ed-b703-d2e322966045"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:58:34.865945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.865891    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt" pod "548be2ce-a087-45ed-b703-d2e322966045" (UID: "548be2ce-a087-45ed-b703-d2e322966045"). InnerVolumeSpecName "kube-api-access-8n6jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:58:34.965062 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.965031    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n6jt\" (UniqueName: \"kubernetes.io/projected/548be2ce-a087-45ed-b703-d2e322966045-kube-api-access-8n6jt\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:34.965062 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.965059    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/548be2ce-a087-45ed-b703-d2e322966045-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:34.965251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:34.965069    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/548be2ce-a087-45ed-b703-d2e322966045-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:58:35.565372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:35.565340    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg" event={"ID":"548be2ce-a087-45ed-b703-d2e322966045","Type":"ContainerDied","Data":"259396d630939776c1a37ae3450eecce64bc09b38dd28510feacd9f3d0ab49fc"}
Nov 01 22:58:35.565372 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:35.565389    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="259396d630939776c1a37ae3450eecce64bc09b38dd28510feacd9f3d0ab49fc"
Nov 01 22:58:35.565684 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:35.565350    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"
Nov 01 22:58:35.587295 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:35.587264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:35.789606 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:35.789574    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:39.490294 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:39.490263    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:40.771349 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:40.771318    4531 scope.go:122] "RemoveContainer" containerID="7710dad263fbc9ff2f0871f3599b5a0b0b68541bee65eccf71db0c2e02940dec"
Nov 01 22:58:40.977212 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:40.977162    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:42.530822 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:58:42.530787    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:58:42.947633 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:42.947599    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:43.145713 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:43.145634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:46.065492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:46.065443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:50.671823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:50.668993    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:50.852488 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:50.852456    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:53.535589 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:53.535550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:55.625317 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:55.625278    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:58:59.410886 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.410854    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"]
Nov 01 22:58:59.411701 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.411683    4531 cpu_manager.go:414] "containerMap: removing container" podUID="548be2ce-a087-45ed-b703-d2e322966045" containerName="revoke-keys"
Nov 01 22:58:59.411759 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.411702    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="548be2ce-a087-45ed-b703-d2e322966045" containerName="revoke-keys"
Nov 01 22:58:59.411856 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.411846    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="548be2ce-a087-45ed-b703-d2e322966045" containerName="revoke-keys"
Nov 01 22:58:59.417050 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.417033    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.519169 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.519139    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:58:59.519385 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.519140    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:58:59.524881 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.524857    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"]
Nov 01 22:58:59.530533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.530498    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.530699 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.530678    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.530855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.530836    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6r2j\" (UniqueName: \"kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.632154 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.632125    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6r2j\" (UniqueName: \"kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.632305 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.632240    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.632351 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.632309    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.632944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.632922    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.635092 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.635053    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.686265 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.686195    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6r2j\" (UniqueName: \"kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j\") pod \"maas-api-revoke-keys-e2e-mt-945efd02-jjrsw\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:58:59.736122 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:58:59.735835    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:59:00.316274 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:00.316249    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"]
Nov 01 22:59:00.318770 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:59:00.318736    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822b205c_3658_44ab_9f88_8a30b4039ba0.slice/crio-a94d84218fd60851c1d2c9f688a1e2e1f8ad8d8d72fc7041943152e39dc8159e WatchSource:0}: Error finding container a94d84218fd60851c1d2c9f688a1e2e1f8ad8d8d72fc7041943152e39dc8159e: Status 404 returned error can't find the container with id a94d84218fd60851c1d2c9f688a1e2e1f8ad8d8d72fc7041943152e39dc8159e
Nov 01 22:59:00.727204 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:00.727169    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:00.944414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:00.944315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw" event={"ID":"822b205c-3658-44ab-9f88-8a30b4039ba0","Type":"ContainerStarted","Data":"917d55bd33860e9d931ddb7ed14d0cc1ebc717551bfd6a1f368c917c85ceb93e"}
Nov 01 22:59:00.944414 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:00.944389    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw" event={"ID":"822b205c-3658-44ab-9f88-8a30b4039ba0","Type":"ContainerStarted","Data":"a94d84218fd60851c1d2c9f688a1e2e1f8ad8d8d72fc7041943152e39dc8159e"}
Nov 01 22:59:01.058373 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:01.058323    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw" podStartSLOduration=2.058308883 podStartE2EDuration="2.058308883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:01.053762278 +0000 UTC m=+2122.371412275" watchObservedRunningTime="2026-11-01 22:59:01.058308883 +0000 UTC m=+2122.375958884"
Nov 01 22:59:01.954936 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:01.954900    4531 generic.go:356] "Generic (PLEG): container finished" podID="822b205c-3658-44ab-9f88-8a30b4039ba0" containerID="917d55bd33860e9d931ddb7ed14d0cc1ebc717551bfd6a1f368c917c85ceb93e" exitCode=0
Nov 01 22:59:01.955495 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:01.954995    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw" event={"ID":"822b205c-3658-44ab-9f88-8a30b4039ba0","Type":"ContainerDied","Data":"917d55bd33860e9d931ddb7ed14d0cc1ebc717551bfd6a1f368c917c85ceb93e"}
Nov 01 22:59:01.990016 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:01.989975    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"]
Nov 01 22:59:01.990253 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:01.990228    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy" containerID="cri-o://6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4" gracePeriod=30
Nov 01 22:59:02.033872 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:02.033825    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"]
Nov 01 22:59:02.034057 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:02.034034    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy" containerID="cri-o://322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696" gracePeriod=30
Nov 01 22:59:03.134341 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.134318    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:59:03.296489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.296394    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca\") pod \"822b205c-3658-44ab-9f88-8a30b4039ba0\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") "
Nov 01 22:59:03.296489 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.296452    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca\") pod \"822b205c-3658-44ab-9f88-8a30b4039ba0\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") "
Nov 01 22:59:03.296757 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.296568    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j\" (UniqueName: \"kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j\") pod \"822b205c-3658-44ab-9f88-8a30b4039ba0\" (UID: \"822b205c-3658-44ab-9f88-8a30b4039ba0\") "
Nov 01 22:59:03.296993 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.296960    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca" pod "822b205c-3658-44ab-9f88-8a30b4039ba0" (UID: "822b205c-3658-44ab-9f88-8a30b4039ba0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:03.297165 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.297145    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:03.298680 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.298650    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca" pod "822b205c-3658-44ab-9f88-8a30b4039ba0" (UID: "822b205c-3658-44ab-9f88-8a30b4039ba0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:59:03.299145 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.299122    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j" pod "822b205c-3658-44ab-9f88-8a30b4039ba0" (UID: "822b205c-3658-44ab-9f88-8a30b4039ba0"). InnerVolumeSpecName "kube-api-access-x6r2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:03.398388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.398349    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6r2j\" (UniqueName: \"kubernetes.io/projected/822b205c-3658-44ab-9f88-8a30b4039ba0-kube-api-access-x6r2j\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:03.398388 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.398385    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822b205c-3658-44ab-9f88-8a30b4039ba0-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:03.973299 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.973269    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"
Nov 01 22:59:03.973491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.973289    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw" event={"ID":"822b205c-3658-44ab-9f88-8a30b4039ba0","Type":"ContainerDied","Data":"a94d84218fd60851c1d2c9f688a1e2e1f8ad8d8d72fc7041943152e39dc8159e"}
Nov 01 22:59:03.973491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:03.973351    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a94d84218fd60851c1d2c9f688a1e2e1f8ad8d8d72fc7041943152e39dc8159e"
Nov 01 22:59:05.951983 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:05.951940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:07.402542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.402513    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:59:07.463228 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.463201    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:59:07.580564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.580439    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.580564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.580509    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.580564 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.580550    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.580892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.580592    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.580892 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.580627    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.582086 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582059    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.582310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582112    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.582310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.580934    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:07.582310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.581258    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.582310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.581964    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.582310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582288    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.582310 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582298    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.582709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582515    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.582709 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582609    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.582904 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582878    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.582977 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.582944    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.583041 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583004    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.583098 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583048    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.583168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583112    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.583168 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583156    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.583286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583186    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.583286 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583266    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy\") pod \"f010fe17-8581-43df-9211-5b59979b9fe7\" (UID: \"f010fe17-8581-43df-9211-5b59979b9fe7\") "
Nov 01 22:59:07.583383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583311    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.583383 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.583344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token\") pod \"495700ae-9b3d-47c4-ad34-3d34aad3a218\" (UID: \"495700ae-9b3d-47c4-ad34-3d34aad3a218\") "
Nov 01 22:59:07.584036 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584012    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f010fe17-8581-43df-9211-5b59979b9fe7-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.584340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584038    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.584340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584052    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.584340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584065    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.584340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584077    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.584340 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584089    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.585284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584686    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "kube-api-access-v8wvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:07.585284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584808    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.585284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584923    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.585284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.584924    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:07.585284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.585058    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:07.585284 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.585177    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.585643 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.585547    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.586428 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.586403    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:59:07.587384 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.587359    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:59:07.587665 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.587645    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "kube-api-access-nlrrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:07.588864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.588832    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy" pod "f010fe17-8581-43df-9211-5b59979b9fe7" (UID: "f010fe17-8581-43df-9211-5b59979b9fe7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:07.589470 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.589449    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token" pod "495700ae-9b3d-47c4-ad34-3d34aad3a218" (UID: "495700ae-9b3d-47c4-ad34-3d34aad3a218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:07.685570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685537    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrrc\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-kube-api-access-nlrrc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685568    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685570 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685578    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495700ae-9b3d-47c4-ad34-3d34aad3a218-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685588    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f010fe17-8581-43df-9211-5b59979b9fe7-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685598    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685608    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685616    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f010fe17-8581-43df-9211-5b59979b9fe7-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685639    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685648    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685664    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f010fe17-8581-43df-9211-5b59979b9fe7-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685673    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wvr\" (UniqueName: \"kubernetes.io/projected/495700ae-9b3d-47c4-ad34-3d34aad3a218-kube-api-access-v8wvr\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:07.685911 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:07.685681    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495700ae-9b3d-47c4-ad34-3d34aad3a218-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:08.020514 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.020483    4531 generic.go:356] "Generic (PLEG): container finished" podID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerID="322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696" exitCode=0
Nov 01 22:59:08.020686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.020542    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"
Nov 01 22:59:08.020686 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.020617    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" event={"ID":"495700ae-9b3d-47c4-ad34-3d34aad3a218","Type":"ContainerDied","Data":"322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696"}
Nov 01 22:59:08.020823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.020689    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7" event={"ID":"495700ae-9b3d-47c4-ad34-3d34aad3a218","Type":"ContainerDied","Data":"44ca55349790eb613f379847a1dc2a2be70c27dccc5970ad22159018db536305"}
Nov 01 22:59:08.020823 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.020719    4531 scope.go:122] "RemoveContainer" containerID="322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696"
Nov 01 22:59:08.022677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.022482    4531 generic.go:356] "Generic (PLEG): container finished" podID="f010fe17-8581-43df-9211-5b59979b9fe7" containerID="6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4" exitCode=0
Nov 01 22:59:08.022677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.022512    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" event={"ID":"f010fe17-8581-43df-9211-5b59979b9fe7","Type":"ContainerDied","Data":"6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4"}
Nov 01 22:59:08.022677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.022578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f" event={"ID":"f010fe17-8581-43df-9211-5b59979b9fe7","Type":"ContainerDied","Data":"52d9aabdc8aa5dde24ff76cf56b69598d780584a7f8d0b81ab944e3c39cd86ad"}
Nov 01 22:59:08.022677 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.022524    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"
Nov 01 22:59:08.033581 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.033561    4531 scope.go:122] "RemoveContainer" containerID="322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696"
Nov 01 22:59:08.033916 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:08.033886    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696\": container with ID starting with 322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696 not found: ID does not exist" containerID="322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696"
Nov 01 22:59:08.034030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.033925    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696"} err="failed to get container status \"322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696\": rpc error: code = NotFound desc = could not find container \"322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696\": container with ID starting with 322302425d1bc83a82f885fb82abc005988c347c263af9251053c576cef85696 not found: ID does not exist"
Nov 01 22:59:08.034030 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.033947    4531 scope.go:122] "RemoveContainer" containerID="6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4"
Nov 01 22:59:08.044949 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.044931    4531 scope.go:122] "RemoveContainer" containerID="6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4"
Nov 01 22:59:08.045212 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:08.045194    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4\": container with ID starting with 6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4 not found: ID does not exist" containerID="6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4"
Nov 01 22:59:08.045252 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.045219    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4"} err="failed to get container status \"6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4\": rpc error: code = NotFound desc = could not find container \"6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4\": container with ID starting with 6d13df5ccb0631185c9033fee485d8aff8cd624891f6358efc1c1d47be92dde4 not found: ID does not exist"
Nov 01 22:59:08.198548 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.198522    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"]
Nov 01 22:59:08.203518 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.203491    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-854q7"]
Nov 01 22:59:08.309629 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.309544    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"]
Nov 01 22:59:08.315189 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.315161    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-9z76f"]
Nov 01 22:59:08.819579 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.819531    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" path="/var/lib/kubelet/pods/495700ae-9b3d-47c4-ad34-3d34aad3a218/volumes"
Nov 01 22:59:08.820211 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:08.820192    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" path="/var/lib/kubelet/pods/f010fe17-8581-43df-9211-5b59979b9fe7/volumes"
Nov 01 22:59:10.892516 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:10.892480    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:11.382314 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:11.380570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:11.578880 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:11.578847    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:12.550950 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:12.550912    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:59:15.015059 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.015027    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"]
Nov 01 22:59:15.015448 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.015218    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" podUID="e7144b91-f7a8-4c94-8582-9f735c01b8a0" containerName="maas-api" containerID="cri-o://77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea" gracePeriod=30
Nov 01 22:59:15.159190 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.159127    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:15.349354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.349325    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:59:15.393318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.393284    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca\") pod \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") "
Nov 01 22:59:15.393533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.393506    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\") pod \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") "
Nov 01 22:59:15.393754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.393659    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs\") pod \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") "
Nov 01 22:59:15.393754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.393714    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca" pod "e7144b91-f7a8-4c94-8582-9f735c01b8a0" (UID: "e7144b91-f7a8-4c94-8582-9f735c01b8a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:15.393923 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.393842    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt\" (UniqueName: \"kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt\") pod \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\" (UID: \"e7144b91-f7a8-4c94-8582-9f735c01b8a0\") "
Nov 01 22:59:15.394518 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.394493    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7144b91-f7a8-4c94-8582-9f735c01b8a0-openshift-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:15.400354 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.400311    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls" pod "e7144b91-f7a8-4c94-8582-9f735c01b8a0" (UID: "e7144b91-f7a8-4c94-8582-9f735c01b8a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:59:15.403223 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.403191    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs" pod "e7144b91-f7a8-4c94-8582-9f735c01b8a0" (UID: "e7144b91-f7a8-4c94-8582-9f735c01b8a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 22:59:15.403678 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.403651    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt" pod "e7144b91-f7a8-4c94-8582-9f735c01b8a0" (UID: "e7144b91-f7a8-4c94-8582-9f735c01b8a0"). InnerVolumeSpecName "kube-api-access-jsjtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:15.496309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.496245    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:15.496309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.496277    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsjtt\" (UniqueName: \"kubernetes.io/projected/e7144b91-f7a8-4c94-8582-9f735c01b8a0-kube-api-access-jsjtt\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:15.496309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:15.496288    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7144b91-f7a8-4c94-8582-9f735c01b8a0-maas-api-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:16.099421 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.099384    4531 generic.go:356] "Generic (PLEG): container finished" podID="e7144b91-f7a8-4c94-8582-9f735c01b8a0" containerID="77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea" exitCode=0
Nov 01 22:59:16.099917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.099443    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"
Nov 01 22:59:16.099917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.099480    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" event={"ID":"e7144b91-f7a8-4c94-8582-9f735c01b8a0","Type":"ContainerDied","Data":"77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea"}
Nov 01 22:59:16.099917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.099540    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22" event={"ID":"e7144b91-f7a8-4c94-8582-9f735c01b8a0","Type":"ContainerDied","Data":"86cf5f05423893bb5ca4c64a848781b50dec9e949d5ccb9cc12e0dc19f24050c"}
Nov 01 22:59:16.099917 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.099575    4531 scope.go:122] "RemoveContainer" containerID="77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea"
Nov 01 22:59:16.112196 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.112119    4531 scope.go:122] "RemoveContainer" containerID="77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea"
Nov 01 22:59:16.112528 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:16.112499    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea\": container with ID starting with 77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea not found: ID does not exist" containerID="77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea"
Nov 01 22:59:16.112620 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.112541    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea"} err="failed to get container status \"77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea\": rpc error: code = NotFound desc = could not find container \"77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea\": container with ID starting with 77222b7390b03159c90948c134802cbba0e3111922ffb042f027ebe67caed6ea not found: ID does not exist"
Nov 01 22:59:16.256049 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.256020    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"]
Nov 01 22:59:16.302042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.302009    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-945efd02-79b65bbb9-6fq22"]
Nov 01 22:59:16.666779 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.666746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:16.817947 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.817918    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7144b91-f7a8-4c94-8582-9f735c01b8a0" path="/var/lib/kubelet/pods/e7144b91-f7a8-4c94-8582-9f735c01b8a0/volumes"
Nov 01 22:59:16.871958 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:16.871932    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:17.063567 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:17.063532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:17.117361 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:17.117328    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"]
Nov 01 22:59:17.162864 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:17.162835    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-945efd02-jjrsw"]
Nov 01 22:59:17.569566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:17.569494    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:18.441423 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:18.441394    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"]
Nov 01 22:59:18.441846 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:18.441608    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" containerID="cri-o://d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1" gracePeriod=30
Nov 01 22:59:18.745486 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:18.745448    4531 patch_prober.go:29] interesting pod/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 22:59:18.745667 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:18.745510    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 22:59:18.820182 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:18.820150    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822b205c-3658-44ab-9f88-8a30b4039ba0" path="/var/lib/kubelet/pods/822b205c-3658-44ab-9f88-8a30b4039ba0/volumes"
Nov 01 22:59:22.041465 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:22.041427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:22.047803 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:22.047767    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"]
Nov 01 22:59:22.092708 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:22.092672    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d59ac06-w9nqc"]
Nov 01 22:59:22.599685 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:22.599655    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:22.818019 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:22.817990    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c" path="/var/lib/kubelet/pods/d13d9553-ea54-4cc6-a5e5-8f48d80a3c6c/volumes"
Nov 01 22:59:23.718060 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.718006    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:59:23.809550 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809515    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.809770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809608    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.809770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809649    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.809770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809677    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.809770 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809720    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.810014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809778    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.810014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809831    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.810014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809899    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.810014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809902    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.810014 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.809971    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.810318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810064    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.810318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810152    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.810318 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810243    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy\") pod \"fef54cec-28ed-4f26-b498-27a0012e2f5c\" (UID: \"fef54cec-28ed-4f26-b498-27a0012e2f5c\") "
Nov 01 22:59:23.810494 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810431    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 22:59:23.810838 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810812    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.810838 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810838    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.811051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810854    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.811051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810868    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fef54cec-28ed-4f26-b498-27a0012e2f5c-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.811051 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.810882    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.812483 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.812455    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "kube-api-access-p4v6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:23.812592 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.812507    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 22:59:23.812592 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.812549    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 22:59:23.812811 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.812787    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy" pod "fef54cec-28ed-4f26-b498-27a0012e2f5c" (UID: "fef54cec-28ed-4f26-b498-27a0012e2f5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 22:59:23.913205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.913173    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.913205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.913201    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4v6n\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-kube-api-access-p4v6n\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.913205 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.913212    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:23.913491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:23.913221    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fef54cec-28ed-4f26-b498-27a0012e2f5c-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 22:59:24.179944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.179858    4531 generic.go:356] "Generic (PLEG): container finished" podID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerID="d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1" exitCode=0
Nov 01 22:59:24.179944 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.179908    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"
Nov 01 22:59:24.180174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.179944    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" event={"ID":"fef54cec-28ed-4f26-b498-27a0012e2f5c","Type":"ContainerDied","Data":"d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1"}
Nov 01 22:59:24.180174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.180009    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj" event={"ID":"fef54cec-28ed-4f26-b498-27a0012e2f5c","Type":"ContainerDied","Data":"dfec9d326a50c950f07abf6483dafdd8de09c09139f639d3345ed85097c64d11"}
Nov 01 22:59:24.180174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.180035    4531 scope.go:122] "RemoveContainer" containerID="d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1"
Nov 01 22:59:24.194669 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.194650    4531 scope.go:122] "RemoveContainer" containerID="d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1"
Nov 01 22:59:24.195159 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:24.195126    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1\": container with ID starting with d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1 not found: ID does not exist" containerID="d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1"
Nov 01 22:59:24.195302 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.195155    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1"} err="failed to get container status \"d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1\": rpc error: code = NotFound desc = could not find container \"d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1\": container with ID starting with d1af3eb82a26655a3fdf1e41364688c780acaafb14c42c153abe0ca1b5fe41f1 not found: ID does not exist"
Nov 01 22:59:24.336855 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.336824    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"]
Nov 01 22:59:24.382608 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.382578    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-e71d55-openshift-default-c7868d7fc-4fcfj"]
Nov 01 22:59:24.818549 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:24.818509    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" path="/var/lib/kubelet/pods/fef54cec-28ed-4f26-b498-27a0012e2f5c/volumes"
Nov 01 22:59:25.315264 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:25.315230    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:25.501866 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:25.501833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:27.069343 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:27.069296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:31.950241 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:31.950200    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:32.048491 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:32.048459    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"]
Nov 01 22:59:32.096972 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:32.096936    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-244c5c8d-j4dqz"]
Nov 01 22:59:32.818780 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:32.818746    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f57916e-eed6-43fd-b5d7-90d347603064" path="/var/lib/kubelet/pods/6f57916e-eed6-43fd-b5d7-90d347603064/volumes"
Nov 01 22:59:36.820330 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:36.820294    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:39.873751 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:39.873199    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:41.341134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341094    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"]
Nov 01 22:59:41.341845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341827    4531 cpu_manager.go:414] "containerMap: removing container" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy"
Nov 01 22:59:41.341845 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341843    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341867    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341873    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341882    4531 cpu_manager.go:414] "containerMap: removing container" podUID="822b205c-3658-44ab-9f88-8a30b4039ba0" containerName="revoke-keys"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341887    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822b205c-3658-44ab-9f88-8a30b4039ba0" containerName="revoke-keys"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341919    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e7144b91-f7a8-4c94-8582-9f735c01b8a0" containerName="maas-api"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341924    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7144b91-f7a8-4c94-8582-9f735c01b8a0" containerName="maas-api"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341932    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.341936    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.342031    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f010fe17-8581-43df-9211-5b59979b9fe7" containerName="istio-proxy"
Nov 01 22:59:41.342034 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.342040    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="822b205c-3658-44ab-9f88-8a30b4039ba0" containerName="revoke-keys"
Nov 01 22:59:41.342539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.342051    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7144b91-f7a8-4c94-8582-9f735c01b8a0" containerName="maas-api"
Nov 01 22:59:41.342539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.342058    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fef54cec-28ed-4f26-b498-27a0012e2f5c" containerName="istio-proxy"
Nov 01 22:59:41.342539 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.342066    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="495700ae-9b3d-47c4-ad34-3d34aad3a218" containerName="istio-proxy"
Nov 01 22:59:41.347656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.347635    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.401134 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.401098    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"]
Nov 01 22:59:41.449749 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.442048    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-7da9ed-serving-cert\""
Nov 01 22:59:41.451123 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.451084    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.451309 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.451292    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.451387 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.451327    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clftc\" (UniqueName: \"kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.451473 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.451445    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.554042 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.554011    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.554239 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.554115    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.554239 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:41.554143    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-serving-cert: secret "maas-api-e2e-inf-a-7da9ed-serving-cert" not found
Nov 01 22:59:41.554239 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:41.554211    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls podName:45266f72-2dae-4659-9fe5-0c11d5668cc7 nodeName:}" failed. No retries permitted until 2026-11-01 22:59:42.054190503 +0000 UTC m=+2163.371840496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls") pod "maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" (UID: "45266f72-2dae-4659-9fe5-0c11d5668cc7") : secret "maas-api-e2e-inf-a-7da9ed-serving-cert" not found
Nov 01 22:59:41.554458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.554146    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clftc\" (UniqueName: \"kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.554458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.554374    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.555348 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.555322    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.557368 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.557347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.594394 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.594312    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:41.617377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.617330    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clftc\" (UniqueName: \"kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:41.803754 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:41.800134    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:42.000458 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:42.000428    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:42.085767 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:42.085243    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:42.090616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:42.090585    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\") pod \"maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:42.280799 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:42.279940    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:42.530542 ip-10-0-130-144 kubenswrapper[4531]: E1101 22:59:42.530506    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 22:59:42.667262 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:42.667233    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"]
Nov 01 22:59:42.669973 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:59:42.669943    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45266f72_2dae_4659_9fe5_0c11d5668cc7.slice/crio-a3c06841c41d195dbd1893d9751b55fd25df50e92774aa27fed2d783ad9dbc01 WatchSource:0}: Error finding container a3c06841c41d195dbd1893d9751b55fd25df50e92774aa27fed2d783ad9dbc01: Status 404 returned error can't find the container with id a3c06841c41d195dbd1893d9751b55fd25df50e92774aa27fed2d783ad9dbc01
Nov 01 22:59:43.350662 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:43.350624    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" event={"ID":"45266f72-2dae-4659-9fe5-0c11d5668cc7","Type":"ContainerStarted","Data":"a3c06841c41d195dbd1893d9751b55fd25df50e92774aa27fed2d783ad9dbc01"}
Nov 01 22:59:44.362928 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:44.362887    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" event={"ID":"45266f72-2dae-4659-9fe5-0c11d5668cc7","Type":"ContainerStarted","Data":"2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115"}
Nov 01 22:59:44.363512 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:44.363482    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:44.476248 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:44.476188    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" podStartSLOduration=2.533381133 podStartE2EDuration="3.476169413s" totalImagesPullingTime="942.78828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:44.470036077 +0000 UTC m=+2165.787686090" watchObservedRunningTime="2026-11-01 22:59:44.476169413 +0000 UTC m=+2165.793819416"
Nov 01 22:59:46.719656 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:46.719620    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:47.397492 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:47.397414    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerID="11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b" exitCode=0
Nov 01 22:59:47.397687 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:47.397490    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" event={"ID":"6c400087-2db5-4bbd-9427-be06e9acbf8f","Type":"ContainerDied","Data":"11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"}
Nov 01 22:59:47.397931 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:47.397915    4531 scope.go:122] "RemoveContainer" containerID="11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"
Nov 01 22:59:47.399945 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:47.399915    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:59:47.400031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:47.400001    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:59:47.400077 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:47.400047    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 22:59:48.409031 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:48.408992    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" event={"ID":"6c400087-2db5-4bbd-9427-be06e9acbf8f","Type":"ContainerStarted","Data":"ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0"}
Nov 01 22:59:48.599231 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:48.599188    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:59:48.600883 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:48.600851    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 22:59:48.601037 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:48.600904    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 22:59:49.599244 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:49.599203    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 22:59:49.599668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:49.599276    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 22:59:50.395783 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:50.394360    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 22:59:50.453585 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:50.453555    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:59:50.606827 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:50.606780    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 22:59:50.607874 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:50.607533    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 22:59:51.599436 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:51.599406    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 22:59:51.599616 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:51.599465    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 22:59:51.616097 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:51.616065    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:52.599328 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:52.599292    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 22:59:52.599572 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:52.599346    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 22:59:53.599174 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:53.599137    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body=
Nov 01 22:59:53.599674 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:53.599194    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused"
Nov 01 22:59:54.604632 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:54.604595    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:59:54.605804 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:54.605775    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 22:59:56.556112 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:56.556080    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:57.977045 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:57.977016    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"]
Nov 01 22:59:57.981941 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:57.981922    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.036542 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.036514    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"]
Nov 01 22:59:58.067377 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.067354    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnrxh\" (UniqueName: \"kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.067566 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.067539    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.067668 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.067648    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.076930 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.076903    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 22:59:58.077017 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.076964    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 22:59:58.169251 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.169228    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.169392 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.169287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.169441 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.169389    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnrxh\" (UniqueName: \"kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.170021 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.169992    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.171671 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.171649    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.223345 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.223321    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnrxh\" (UniqueName: \"kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh\") pod \"maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.296209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.296155    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 22:59:58.681175 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:58.681153    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"]
Nov 01 22:59:58.684232 ip-10-0-130-144 kubenswrapper[4531]: W1101 22:59:58.684194    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86326a5d_e4f1_4038_9a92_453fa427443a.slice/crio-87e8f4b624ce7d466de70d631c264b2e0f19b1fe95d5f9a3701ecf12fb041d9a WatchSource:0}: Error finding container 87e8f4b624ce7d466de70d631c264b2e0f19b1fe95d5f9a3701ecf12fb041d9a: Status 404 returned error can't find the container with id 87e8f4b624ce7d466de70d631c264b2e0f19b1fe95d5f9a3701ecf12fb041d9a
Nov 01 22:59:59.407209 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:59.407116    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 22:59:59.519533 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:59.519498    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz" event={"ID":"86326a5d-e4f1-4038-9a92-453fa427443a","Type":"ContainerStarted","Data":"eb3cb963c68f49477c65571173d088c6864df9b9dd0af0cebb3010162e594a4a"}
Nov 01 22:59:59.519716 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:59.519584    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz" event={"ID":"86326a5d-e4f1-4038-9a92-453fa427443a","Type":"ContainerStarted","Data":"87e8f4b624ce7d466de70d631c264b2e0f19b1fe95d5f9a3701ecf12fb041d9a"}
Nov 01 22:59:59.628124 ip-10-0-130-144 kubenswrapper[4531]: I1101 22:59:59.628075    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz" podStartSLOduration=2.628062432 podStartE2EDuration="2.628062432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 22:59:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 22:59:59.627175477 +0000 UTC m=+2180.944825479" watchObservedRunningTime="2026-11-01 22:59:59.628062432 +0000 UTC m=+2180.945712432"
Nov 01 23:00:00.182606 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.182567    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"]
Nov 01 23:00:00.187712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.187685    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"]
Nov 01 23:00:00.187923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.187826    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.191901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.191879    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"]
Nov 01 23:00:00.192014 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.191985    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.196004 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.195985    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.214122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.214101    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf85t\" (UniqueName: \"kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.214258 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.214136    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.214258 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.214213    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tm7\" (UniqueName: \"kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.214379 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.214268    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.214437 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.214409    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f87dc\" (UniqueName: \"kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.214437 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.214431    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.225107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.225084    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"]
Nov 01 23:00:00.229076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.229061    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.241560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.241540    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"]
Nov 01 23:00:00.245551 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.245533    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.252555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.252538    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"]
Nov 01 23:00:00.257639 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.257616    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"]
Nov 01 23:00:00.287882 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.287857    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"]
Nov 01 23:00:00.293551 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.293525    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.294064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.294043    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"]
Nov 01 23:00:00.307425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.300282    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"]
Nov 01 23:00:00.307425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.300375    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.307425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.306547    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"]
Nov 01 23:00:00.311289 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.311259    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"]
Nov 01 23:00:00.315246 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315222    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chbwg\" (UniqueName: \"kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg\") pod \"maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.315366 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thnjj\" (UniqueName: \"kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj\") pod \"maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.315366 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315290    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.315486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315367    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.315486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315393    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.315486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315417    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.315656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315549    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf85t\" (UniqueName: \"kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.315656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315595    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.315796 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315657    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvh8t\" (UniqueName: \"kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.315796 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315684    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tm7\" (UniqueName: \"kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.315796 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315712    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.315968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315905    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f87dc\" (UniqueName: \"kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.315968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315925    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhs4s\" (UniqueName: \"kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s\") pod \"maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.315968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.315949    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.316371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.316347    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.316548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.316523    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.316648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.316607    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.316648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.316630    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.336702 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.336679    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"]
Nov 01 23:00:00.365173 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.364492    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"]
Nov 01 23:00:00.371975 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.371948    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"]
Nov 01 23:00:00.379993 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.379948    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"]
Nov 01 23:00:00.390693 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.389922    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.396748 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.394607    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"]
Nov 01 23:00:00.404773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.399092    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf85t\" (UniqueName: \"kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.404773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.401327    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f87dc\" (UniqueName: \"kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.408998 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.408974    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tm7\" (UniqueName: \"kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.415497 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.415473    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"]
Nov 01 23:00:00.417592 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417563    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.417715 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.417798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417747    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhs4s\" (UniqueName: \"kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s\") pod \"maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.417863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417795    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6lhb\" (UniqueName: \"kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.417863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417844    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chbwg\" (UniqueName: \"kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg\") pod \"maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.417971 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417871    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9mzq\" (UniqueName: \"kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.417971 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417901    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thnjj\" (UniqueName: \"kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj\") pod \"maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.417971 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.418125 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.417985    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.418125 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.418012    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.418125 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.418044    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.418283 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.418158    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvh8t\" (UniqueName: \"kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.419861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.419146    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.419861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.419169    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.419861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.419814    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.419861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.419814    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.432064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.432028    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"]
Nov 01 23:00:00.432192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.432109    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.445462 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.445233    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"]
Nov 01 23:00:00.450617 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.450598    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.455455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.455432    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"]
Nov 01 23:00:00.461471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.461442    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.468709 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.468675    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"]
Nov 01 23:00:00.488577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.488550    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thnjj\" (UniqueName: \"kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj\") pod \"maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.488767 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.488746    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvh8t\" (UniqueName: \"kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.489989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.489972    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhs4s\" (UniqueName: \"kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s\") pod \"maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.490113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.490090    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chbwg\" (UniqueName: \"kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg\") pod \"maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.512787 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.505080    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:00.514003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.513982    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:00.514232 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.514220    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"]
Nov 01 23:00:00.520203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.520181    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cjgg\" (UniqueName: \"kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.520395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.520376    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.520519 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.520503    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgrt\" (UniqueName: \"kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.520701 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.520686    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.521700 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.521680    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.523938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.523914    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6lhb\" (UniqueName: \"kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.524086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.524071    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9mzq\" (UniqueName: \"kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.524219 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.524205    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.524407 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.524390    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.524529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.524513    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd2b9\" (UniqueName: \"kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9\") pod \"maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.526874 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.521456    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.527837 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.527812    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.527969 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.521615    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:00.547547 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.544803    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:00.547547 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.546615    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"]
Nov 01 23:00:00.573439 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.570810    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:00.573439 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.573266    4531 generic.go:356] "Generic (PLEG): container finished" podID="86326a5d-e4f1-4038-9a92-453fa427443a" containerID="eb3cb963c68f49477c65571173d088c6864df9b9dd0af0cebb3010162e594a4a" exitCode=0
Nov 01 23:00:00.573439 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.573400    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz" event={"ID":"86326a5d-e4f1-4038-9a92-453fa427443a","Type":"ContainerDied","Data":"eb3cb963c68f49477c65571173d088c6864df9b9dd0af0cebb3010162e594a4a"}
Nov 01 23:00:00.583569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.583427    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9mzq\" (UniqueName: \"kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq\") pod \"maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.584511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.584489    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6lhb\" (UniqueName: \"kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb\") pod \"maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.616071 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.614037    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.632916    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.637293    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.640003    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cjgg\" (UniqueName: \"kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.640084    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgrt\" (UniqueName: \"kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.640683    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.640804    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.640903    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.641161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.640938    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd2b9\" (UniqueName: \"kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9\") pod \"maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.643490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.642031    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.643490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.642527    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.643490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.643138    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.724629 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.716994    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd2b9\" (UniqueName: \"kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9\") pod \"maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.756198 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.756168    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cjgg\" (UniqueName: \"kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.758794 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.758697    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgrt\" (UniqueName: \"kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.769290 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.769069    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:00.801527 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.801364    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:00.827744 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.827661    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:00.850129 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.849953    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:00.867964 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.861648    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:00.935995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:00.935467    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"]
Nov 01 23:00:00.939706 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:00.939677    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b0859f_d187_4896_9e50_f9d981010d7d.slice/crio-e18a0ee605507d07a9d977fab4e8f1b1be019530d095b0311c5253a239fadbf7 WatchSource:0}: Error finding container e18a0ee605507d07a9d977fab4e8f1b1be019530d095b0311c5253a239fadbf7: Status 404 returned error can't find the container with id e18a0ee605507d07a9d977fab4e8f1b1be019530d095b0311c5253a239fadbf7
Nov 01 23:00:01.167130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.167106    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"]
Nov 01 23:00:01.169830 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:01.169797    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec531e1a_fb93_4b56_afed_7a77461092cd.slice/crio-cb3a79843e1be2e5a393b7e334406f3dd09f27e672b916225b9d39978807e34b WatchSource:0}: Error finding container cb3a79843e1be2e5a393b7e334406f3dd09f27e672b916225b9d39978807e34b: Status 404 returned error can't find the container with id cb3a79843e1be2e5a393b7e334406f3dd09f27e672b916225b9d39978807e34b
Nov 01 23:00:01.250806 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:01.250760    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc727fe96_45a3_4928_b0b0_20a52737f04b.slice/crio-c641d345be3fa399205641b53421ae11c70d8072c9f6ae967539d912b4652d6d WatchSource:0}: Error finding container c641d345be3fa399205641b53421ae11c70d8072c9f6ae967539d912b4652d6d: Status 404 returned error can't find the container with id c641d345be3fa399205641b53421ae11c70d8072c9f6ae967539d912b4652d6d
Nov 01 23:00:01.254603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.253575    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"]
Nov 01 23:00:01.256900 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:01.256875    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6ffee94_ed10_4a3a_b3fb_856cfc4b36a7.slice/crio-2d08f49a3781e4635556fd0d4f5569240dd971cb1970fb17b3aa611856efa3ed WatchSource:0}: Error finding container 2d08f49a3781e4635556fd0d4f5569240dd971cb1970fb17b3aa611856efa3ed: Status 404 returned error can't find the container with id 2d08f49a3781e4635556fd0d4f5569240dd971cb1970fb17b3aa611856efa3ed
Nov 01 23:00:01.258400 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.258364    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"]
Nov 01 23:00:01.271915 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.271891    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"]
Nov 01 23:00:01.293604 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.293577    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"]
Nov 01 23:00:01.348943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.348902    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"]
Nov 01 23:00:01.354314 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.354289    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"]
Nov 01 23:00:01.359119 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.359094    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"]
Nov 01 23:00:01.392517 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.392475    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"]
Nov 01 23:00:01.516775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.516631    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"]
Nov 01 23:00:01.584599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.584574    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerStarted","Data":"bf72f045e7edc5ed9640ea2590de1089b1d9325ec878da32dc8f90bc93f04a44"}
Nov 01 23:00:01.586257 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.586233    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerStarted","Data":"2d08f49a3781e4635556fd0d4f5569240dd971cb1970fb17b3aa611856efa3ed"}
Nov 01 23:00:01.587740 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.587704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerStarted","Data":"19f50cace0bd9aa115dccef5c2165c244a14c8080103dd0f1e165582cd70e391"}
Nov 01 23:00:01.589496 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.589466    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" event={"ID":"39b0859f-d187-4896-9e50-f9d981010d7d","Type":"ContainerStarted","Data":"e18a0ee605507d07a9d977fab4e8f1b1be019530d095b0311c5253a239fadbf7"}
Nov 01 23:00:01.590782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.590757    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerStarted","Data":"e4b314dcfdd0bbeca29676825f4bd8787bdd6877f570d64510d59204064d6655"}
Nov 01 23:00:01.592378 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.592357    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerStarted","Data":"c641d345be3fa399205641b53421ae11c70d8072c9f6ae967539d912b4652d6d"}
Nov 01 23:00:01.593921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.593890    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerStarted","Data":"50e5e923673e8901b55bb973a651318dea9eaa142237649b8d5028d66e3773ef"}
Nov 01 23:00:01.595492 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.595466    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" event={"ID":"aec38feb-bb0d-49e6-bb55-5a5693ccca4d","Type":"ContainerStarted","Data":"be2fb5ca7ac7aeb0b4b76387cd6da6be1750316c5793dcdbcbc97c05ba720f3c"}
Nov 01 23:00:01.596832 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.596806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerStarted","Data":"119e5efe5a645d467a2605f5d3f4c8e278aaebabda40be54e93b497f7602a220"}
Nov 01 23:00:01.598140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.598117    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" event={"ID":"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2","Type":"ContainerStarted","Data":"3f00bd98c989a2af39dd8a427e13b665e0f49a510df21963a1df88c9f7dacfca"}
Nov 01 23:00:01.599572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.599550    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerStarted","Data":"cb3a79843e1be2e5a393b7e334406f3dd09f27e672b916225b9d39978807e34b"}
Nov 01 23:00:01.599657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.599597    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"]
Nov 01 23:00:01.601311 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:01.601283    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc33a895_3e28_4ab8_9e42_7076c0f9380d.slice/crio-e0bacc1e55bd214336cf50cb84065d502ccc0abfa9bc89d005394766e86b6797 WatchSource:0}: Error finding container e0bacc1e55bd214336cf50cb84065d502ccc0abfa9bc89d005394766e86b6797: Status 404 returned error can't find the container with id e0bacc1e55bd214336cf50cb84065d502ccc0abfa9bc89d005394766e86b6797
Nov 01 23:00:01.757495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.756560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:01.789755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.789683    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" podStartSLOduration=1.7896623329999999 podStartE2EDuration="1.789662333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:01.78894073 +0000 UTC m=+2183.106590728" watchObservedRunningTime="2026-11-01 23:00:01.789662333 +0000 UTC m=+2183.107312335"
Nov 01 23:00:01.969899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:01.969872    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 23:00:02.070982 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.070895    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca\") pod \"86326a5d-e4f1-4038-9a92-453fa427443a\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") "
Nov 01 23:00:02.071511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.071473    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca\") pod \"86326a5d-e4f1-4038-9a92-453fa427443a\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") "
Nov 01 23:00:02.071609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.071593    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh\" (UniqueName: \"kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh\") pod \"86326a5d-e4f1-4038-9a92-453fa427443a\" (UID: \"86326a5d-e4f1-4038-9a92-453fa427443a\") "
Nov 01 23:00:02.071861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.071837    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca" pod "86326a5d-e4f1-4038-9a92-453fa427443a" (UID: "86326a5d-e4f1-4038-9a92-453fa427443a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:02.072188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.072172    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:02.166399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.166342    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca" pod "86326a5d-e4f1-4038-9a92-453fa427443a" (UID: "86326a5d-e4f1-4038-9a92-453fa427443a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:02.166573 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.166471    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh" pod "86326a5d-e4f1-4038-9a92-453fa427443a" (UID: "86326a5d-e4f1-4038-9a92-453fa427443a"). InnerVolumeSpecName "kube-api-access-pnrxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:02.173631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.173607    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86326a5d-e4f1-4038-9a92-453fa427443a-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:02.173766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.173634    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnrxh\" (UniqueName: \"kubernetes.io/projected/86326a5d-e4f1-4038-9a92-453fa427443a-kube-api-access-pnrxh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:02.618515 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.618428    4531 generic.go:356] "Generic (PLEG): container finished" podID="39b0859f-d187-4896-9e50-f9d981010d7d" containerID="5edb4f1f73f0a0970b4dfc3443055ea1bef3ab06ebe90ac91311cb4e390b522e" exitCode=60
Nov 01 23:00:02.618967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.618519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" event={"ID":"39b0859f-d187-4896-9e50-f9d981010d7d","Type":"ContainerDied","Data":"5edb4f1f73f0a0970b4dfc3443055ea1bef3ab06ebe90ac91311cb4e390b522e"}
Nov 01 23:00:02.618967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.618899    4531 scope.go:122] "RemoveContainer" containerID="5edb4f1f73f0a0970b4dfc3443055ea1bef3ab06ebe90ac91311cb4e390b522e"
Nov 01 23:00:02.620560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.620497    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerStarted","Data":"cf445c8d33e2109d75ee500807033e383eee45a11cd3bccb13a96be3dca11017"}
Nov 01 23:00:02.622819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.622795    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerStarted","Data":"e26b800bb25ebbb2ad6e24f574322cf069c6a3dfd2d13cb8cb3215310b3d6262"}
Nov 01 23:00:02.624803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.624772    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerStarted","Data":"7b4b50aec381d6ec96b17ae41f3af0f018c2b01341878b45f0c3d5d0f8d0f7a7"}
Nov 01 23:00:02.626799 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.626770    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" event={"ID":"aec38feb-bb0d-49e6-bb55-5a5693ccca4d","Type":"ContainerStarted","Data":"3469d491ea1675f2b60ffbe8dc47a1c9e56b8cd8b2d1e1aa92a81eecb274c893"}
Nov 01 23:00:02.628526 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.628500    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerStarted","Data":"6504bde4bcaf24b7dacb56e77faf32b13b70eab2018daa796235c84fe4b16fad"}
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.630158    4531 generic.go:356] "Generic (PLEG): container finished" podID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerID="fe360a57c60f1b10569bc1aa56697c7be3f2b44691bbe18f8ca59b280c455361" exitCode=60
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.630240    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerDied","Data":"fe360a57c60f1b10569bc1aa56697c7be3f2b44691bbe18f8ca59b280c455361"}
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.630435    4531 scope.go:122] "RemoveContainer" containerID="fe360a57c60f1b10569bc1aa56697c7be3f2b44691bbe18f8ca59b280c455361"
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.631905    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" event={"ID":"bc33a895-3e28-4ab8-9e42-7076c0f9380d","Type":"ContainerStarted","Data":"e0bacc1e55bd214336cf50cb84065d502ccc0abfa9bc89d005394766e86b6797"}
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.633578    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.633580    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz" event={"ID":"86326a5d-e4f1-4038-9a92-453fa427443a","Type":"ContainerDied","Data":"87e8f4b624ce7d466de70d631c264b2e0f19b1fe95d5f9a3701ecf12fb041d9a"}
Nov 01 23:00:02.636352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.633622    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87e8f4b624ce7d466de70d631c264b2e0f19b1fe95d5f9a3701ecf12fb041d9a"
Nov 01 23:00:02.643862 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.643834    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerStarted","Data":"52e61179a8db328e5e7279251fc4d04b59b47e049136137f262f1ff22948f105"}
Nov 01 23:00:02.645840 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.645815    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerStarted","Data":"ec6fe7173537cfc34f78a2c136965eef13688a050f4f7d20feb6d32c2d6b1859"}
Nov 01 23:00:02.647788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.647765    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerStarted","Data":"5ab5e2b0691f2659de3575e2fe79e267e1a39f9083590b6e8d626ecbe366a4a5"}
Nov 01 23:00:02.836509 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.836458    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" podStartSLOduration=2.836444393 podStartE2EDuration="2.836444393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.832257697 +0000 UTC m=+2184.149907698" watchObservedRunningTime="2026-11-01 23:00:02.836444393 +0000 UTC m=+2184.154094397"
Nov 01 23:00:02.990925 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:02.990876    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" podStartSLOduration=2.990859264 podStartE2EDuration="2.990859264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:02.989790376 +0000 UTC m=+2184.307440384" watchObservedRunningTime="2026-11-01 23:00:02.990859264 +0000 UTC m=+2184.308509265"
Nov 01 23:00:03.096490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.096442    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" podStartSLOduration=3.096427856 podStartE2EDuration="3.096427856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.095586495 +0000 UTC m=+2184.413236496" watchObservedRunningTime="2026-11-01 23:00:03.096427856 +0000 UTC m=+2184.414077857"
Nov 01 23:00:03.207480 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.207415    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" podStartSLOduration=3.207397494 podStartE2EDuration="3.207397494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.203491985 +0000 UTC m=+2184.521141986" watchObservedRunningTime="2026-11-01 23:00:03.207397494 +0000 UTC m=+2184.525047497"
Nov 01 23:00:03.313508 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.313444    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" podStartSLOduration=3.313425489 podStartE2EDuration="3.313425489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.311653692 +0000 UTC m=+2184.629303692" watchObservedRunningTime="2026-11-01 23:00:03.313425489 +0000 UTC m=+2184.631075489"
Nov 01 23:00:03.419140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.419090    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" podStartSLOduration=3.419075417 podStartE2EDuration="3.419075417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.417416541 +0000 UTC m=+2184.735066542" watchObservedRunningTime="2026-11-01 23:00:03.419075417 +0000 UTC m=+2184.736725417"
Nov 01 23:00:03.526595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.526538    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" podStartSLOduration=3.526524764 podStartE2EDuration="3.526524764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.524635181 +0000 UTC m=+2184.842285182" watchObservedRunningTime="2026-11-01 23:00:03.526524764 +0000 UTC m=+2184.844174764"
Nov 01 23:00:03.659575 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.659535    4531 generic.go:356] "Generic (PLEG): container finished" podID="99225305-c4f2-4987-9fcd-833f501913a8" containerID="cf445c8d33e2109d75ee500807033e383eee45a11cd3bccb13a96be3dca11017" exitCode=60
Nov 01 23:00:03.660002 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.659618    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerDied","Data":"cf445c8d33e2109d75ee500807033e383eee45a11cd3bccb13a96be3dca11017"}
Nov 01 23:00:03.660002 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.659871    4531 scope.go:122] "RemoveContainer" containerID="cf445c8d33e2109d75ee500807033e383eee45a11cd3bccb13a96be3dca11017"
Nov 01 23:00:03.661796 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.661768    4531 generic.go:356] "Generic (PLEG): container finished" podID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerID="e26b800bb25ebbb2ad6e24f574322cf069c6a3dfd2d13cb8cb3215310b3d6262" exitCode=60
Nov 01 23:00:03.661911 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.661866    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerDied","Data":"e26b800bb25ebbb2ad6e24f574322cf069c6a3dfd2d13cb8cb3215310b3d6262"}
Nov 01 23:00:03.662056 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.662034    4531 scope.go:122] "RemoveContainer" containerID="e26b800bb25ebbb2ad6e24f574322cf069c6a3dfd2d13cb8cb3215310b3d6262"
Nov 01 23:00:03.664052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.664035    4531 generic.go:356] "Generic (PLEG): container finished" podID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerID="7b4b50aec381d6ec96b17ae41f3af0f018c2b01341878b45f0c3d5d0f8d0f7a7" exitCode=60
Nov 01 23:00:03.664144 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.664115    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerDied","Data":"7b4b50aec381d6ec96b17ae41f3af0f018c2b01341878b45f0c3d5d0f8d0f7a7"}
Nov 01 23:00:03.664321 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.664303    4531 scope.go:122] "RemoveContainer" containerID="7b4b50aec381d6ec96b17ae41f3af0f018c2b01341878b45f0c3d5d0f8d0f7a7"
Nov 01 23:00:03.666062 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.666042    4531 generic.go:356] "Generic (PLEG): container finished" podID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerID="3469d491ea1675f2b60ffbe8dc47a1c9e56b8cd8b2d1e1aa92a81eecb274c893" exitCode=60
Nov 01 23:00:03.666127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.666111    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" event={"ID":"aec38feb-bb0d-49e6-bb55-5a5693ccca4d","Type":"ContainerDied","Data":"3469d491ea1675f2b60ffbe8dc47a1c9e56b8cd8b2d1e1aa92a81eecb274c893"}
Nov 01 23:00:03.666342 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.666327    4531 scope.go:122] "RemoveContainer" containerID="3469d491ea1675f2b60ffbe8dc47a1c9e56b8cd8b2d1e1aa92a81eecb274c893"
Nov 01 23:00:03.668054 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.668032    4531 generic.go:356] "Generic (PLEG): container finished" podID="086dc50c-8e76-437f-af84-ca77b63d7228" containerID="6504bde4bcaf24b7dacb56e77faf32b13b70eab2018daa796235c84fe4b16fad" exitCode=60
Nov 01 23:00:03.668147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.668111    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerDied","Data":"6504bde4bcaf24b7dacb56e77faf32b13b70eab2018daa796235c84fe4b16fad"}
Nov 01 23:00:03.668306 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.668291    4531 scope.go:122] "RemoveContainer" containerID="6504bde4bcaf24b7dacb56e77faf32b13b70eab2018daa796235c84fe4b16fad"
Nov 01 23:00:03.669910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.669892    4531 generic.go:356] "Generic (PLEG): container finished" podID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerID="04c42daee3ef811a57d5fd4bddf022305fff2d1b5dc7ad6769fe9b3a65b947ea" exitCode=60
Nov 01 23:00:03.669998 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.669977    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" event={"ID":"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2","Type":"ContainerDied","Data":"04c42daee3ef811a57d5fd4bddf022305fff2d1b5dc7ad6769fe9b3a65b947ea"}
Nov 01 23:00:03.670145 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.670133    4531 scope.go:122] "RemoveContainer" containerID="04c42daee3ef811a57d5fd4bddf022305fff2d1b5dc7ad6769fe9b3a65b947ea"
Nov 01 23:00:03.672181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.672122    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerStarted","Data":"6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d"}
Nov 01 23:00:03.674247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.674139    4531 generic.go:356] "Generic (PLEG): container finished" podID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerID="1d62d17cdb42bf4d3dcdf0cf3ba50d42d1c3a2153beef737a2870ff8cc2b3f07" exitCode=60
Nov 01 23:00:03.674542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.674517    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" event={"ID":"bc33a895-3e28-4ab8-9e42-7076c0f9380d","Type":"ContainerDied","Data":"1d62d17cdb42bf4d3dcdf0cf3ba50d42d1c3a2153beef737a2870ff8cc2b3f07"}
Nov 01 23:00:03.674712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.674676    4531 scope.go:122] "RemoveContainer" containerID="1d62d17cdb42bf4d3dcdf0cf3ba50d42d1c3a2153beef737a2870ff8cc2b3f07"
Nov 01 23:00:03.681685 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.681656    4531 generic.go:356] "Generic (PLEG): container finished" podID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerID="52e61179a8db328e5e7279251fc4d04b59b47e049136137f262f1ff22948f105" exitCode=60
Nov 01 23:00:03.681801 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.681772    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerDied","Data":"52e61179a8db328e5e7279251fc4d04b59b47e049136137f262f1ff22948f105"}
Nov 01 23:00:03.682180 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.682080    4531 scope.go:122] "RemoveContainer" containerID="52e61179a8db328e5e7279251fc4d04b59b47e049136137f262f1ff22948f105"
Nov 01 23:00:03.684557 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.684529    4531 generic.go:356] "Generic (PLEG): container finished" podID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerID="ec6fe7173537cfc34f78a2c136965eef13688a050f4f7d20feb6d32c2d6b1859" exitCode=60
Nov 01 23:00:03.684664 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.684630    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerDied","Data":"ec6fe7173537cfc34f78a2c136965eef13688a050f4f7d20feb6d32c2d6b1859"}
Nov 01 23:00:03.684989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.684969    4531 scope.go:122] "RemoveContainer" containerID="ec6fe7173537cfc34f78a2c136965eef13688a050f4f7d20feb6d32c2d6b1859"
Nov 01 23:00:03.690923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.690900    4531 generic.go:356] "Generic (PLEG): container finished" podID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerID="5ab5e2b0691f2659de3575e2fe79e267e1a39f9083590b6e8d626ecbe366a4a5" exitCode=60
Nov 01 23:00:03.691302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.691199    4531 scope.go:122] "RemoveContainer" containerID="5ab5e2b0691f2659de3575e2fe79e267e1a39f9083590b6e8d626ecbe366a4a5"
Nov 01 23:00:03.691456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.691438    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerDied","Data":"5ab5e2b0691f2659de3575e2fe79e267e1a39f9083590b6e8d626ecbe366a4a5"}
Nov 01 23:00:03.696457 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.696431    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" event={"ID":"39b0859f-d187-4896-9e50-f9d981010d7d","Type":"ContainerStarted","Data":"50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4"}
Nov 01 23:00:03.745416 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:03.745365    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" podStartSLOduration=3.745343652 podStartE2EDuration="3.745343652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:03.741764274 +0000 UTC m=+2185.059414277" watchObservedRunningTime="2026-11-01 23:00:03.745343652 +0000 UTC m=+2185.062993663"
Nov 01 23:00:04.390968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.390917    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" podStartSLOduration=4.390901568 podStartE2EDuration="4.390901568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:04.38967095 +0000 UTC m=+2185.707320951" watchObservedRunningTime="2026-11-01 23:00:04.390901568 +0000 UTC m=+2185.708551609"
Nov 01 23:00:04.707788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.707717    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerStarted","Data":"7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca"}
Nov 01 23:00:04.709914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.709885    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerStarted","Data":"f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109"}
Nov 01 23:00:04.712003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.711979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerStarted","Data":"fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a"}
Nov 01 23:00:04.714017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.713988    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" event={"ID":"aec38feb-bb0d-49e6-bb55-5a5693ccca4d","Type":"ContainerStarted","Data":"feaa89ee8271532ec7ffca3b77e1c7ffc4a70144a2b0ff43e62300f62ded168d"}
Nov 01 23:00:04.715926 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.715898    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerStarted","Data":"d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a"}
Nov 01 23:00:04.718441 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.718254    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" event={"ID":"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2","Type":"ContainerStarted","Data":"5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919"}
Nov 01 23:00:04.721192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.721116    4531 generic.go:356] "Generic (PLEG): container finished" podID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerID="6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d" exitCode=60
Nov 01 23:00:04.721192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.721184    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerDied","Data":"6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d"}
Nov 01 23:00:04.721317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.721220    4531 scope.go:122] "RemoveContainer" containerID="fe360a57c60f1b10569bc1aa56697c7be3f2b44691bbe18f8ca59b280c455361"
Nov 01 23:00:04.721576 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.721557    4531 scope.go:122] "RemoveContainer" containerID="6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d"
Nov 01 23:00:04.721875 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:04.721817    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2_odh-ai-gateway-infra(ec531e1a-fb93-4b56-afed-7a77461092cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd"
Nov 01 23:00:04.724691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.724663    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" event={"ID":"bc33a895-3e28-4ab8-9e42-7076c0f9380d","Type":"ContainerStarted","Data":"684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3"}
Nov 01 23:00:04.726680 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.726646    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerStarted","Data":"a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c"}
Nov 01 23:00:04.729373 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.729352    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerStarted","Data":"d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4"}
Nov 01 23:00:04.731410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.731390    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerStarted","Data":"ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac"}
Nov 01 23:00:04.733442 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.733419    4531 generic.go:356] "Generic (PLEG): container finished" podID="39b0859f-d187-4896-9e50-f9d981010d7d" containerID="50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4" exitCode=60
Nov 01 23:00:04.733522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.733454    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" event={"ID":"39b0859f-d187-4896-9e50-f9d981010d7d","Type":"ContainerDied","Data":"50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4"}
Nov 01 23:00:04.733786 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.733767    4531 scope.go:122] "RemoveContainer" containerID="50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4"
Nov 01 23:00:04.734066 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:04.734043    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj_odh-ai-gateway-infra(39b0859f-d187-4896-9e50-f9d981010d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" podUID="39b0859f-d187-4896-9e50-f9d981010d7d"
Nov 01 23:00:04.738770 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:04.738745    4531 scope.go:122] "RemoveContainer" containerID="5edb4f1f73f0a0970b4dfc3443055ea1bef3ab06ebe90ac91311cb4e390b522e"
Nov 01 23:00:05.264398 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.264351    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" podStartSLOduration=5.264338282 podStartE2EDuration="5.264338282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:05.263326953 +0000 UTC m=+2186.580976954" watchObservedRunningTime="2026-11-01 23:00:05.264338282 +0000 UTC m=+2186.581988283"
Nov 01 23:00:05.591250 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.591202    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" podStartSLOduration=5.591188132 podStartE2EDuration="5.591188132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:05.589470629 +0000 UTC m=+2186.907120631" watchObservedRunningTime="2026-11-01 23:00:05.591188132 +0000 UTC m=+2186.908838133"
Nov 01 23:00:05.744811 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.744783    4531 generic.go:356] "Generic (PLEG): container finished" podID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerID="a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c" exitCode=60
Nov 01 23:00:05.745280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.744865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerDied","Data":"a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c"}
Nov 01 23:00:05.745280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.744931    4531 scope.go:122] "RemoveContainer" containerID="52e61179a8db328e5e7279251fc4d04b59b47e049136137f262f1ff22948f105"
Nov 01 23:00:05.745280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.745127    4531 scope.go:122] "RemoveContainer" containerID="a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c"
Nov 01 23:00:05.745487 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.745464    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx_odh-ai-gateway-infra(51b613b6-48e0-45ce-8cff-9daf0cecc43f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f"
Nov 01 23:00:05.752458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.752433    4531 generic.go:356] "Generic (PLEG): container finished" podID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerID="d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4" exitCode=60
Nov 01 23:00:05.752591 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.752518    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerDied","Data":"d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4"}
Nov 01 23:00:05.753132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.753101    4531 scope.go:122] "RemoveContainer" containerID="d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4"
Nov 01 23:00:05.753642 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.753614    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf_odh-ai-gateway-infra(e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7"
Nov 01 23:00:05.755073 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.755052    4531 generic.go:356] "Generic (PLEG): container finished" podID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerID="ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac" exitCode=60
Nov 01 23:00:05.755167 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.755136    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerDied","Data":"ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac"}
Nov 01 23:00:05.755377 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.755360    4531 scope.go:122] "RemoveContainer" containerID="ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac"
Nov 01 23:00:05.755653 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.755624    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7_odh-ai-gateway-infra(980b4aa0-cf34-4e16-b97f-417bc731da72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72"
Nov 01 23:00:05.756981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.756966    4531 scope.go:122] "RemoveContainer" containerID="50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4"
Nov 01 23:00:05.757245 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.757226    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj_odh-ai-gateway-infra(39b0859f-d187-4896-9e50-f9d981010d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" podUID="39b0859f-d187-4896-9e50-f9d981010d7d"
Nov 01 23:00:05.758826 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.758802    4531 generic.go:356] "Generic (PLEG): container finished" podID="99225305-c4f2-4987-9fcd-833f501913a8" containerID="7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca" exitCode=60
Nov 01 23:00:05.758917 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.758869    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerDied","Data":"7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca"}
Nov 01 23:00:05.759160 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.759111    4531 scope.go:122] "RemoveContainer" containerID="7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca"
Nov 01 23:00:05.759399 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.759356    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq_odh-ai-gateway-infra(99225305-c4f2-4987-9fcd-833f501913a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" podUID="99225305-c4f2-4987-9fcd-833f501913a8"
Nov 01 23:00:05.761109 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.761090    4531 generic.go:356] "Generic (PLEG): container finished" podID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerID="f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109" exitCode=60
Nov 01 23:00:05.761188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.761116    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerDied","Data":"f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109"}
Nov 01 23:00:05.761419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.761402    4531 scope.go:122] "RemoveContainer" containerID="f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109"
Nov 01 23:00:05.761686 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.761666    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p_odh-ai-gateway-infra(c727fe96-45a3-4928-b0b0-20a52737f04b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b"
Nov 01 23:00:05.763388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.763373    4531 scope.go:122] "RemoveContainer" containerID="ec6fe7173537cfc34f78a2c136965eef13688a050f4f7d20feb6d32c2d6b1859"
Nov 01 23:00:05.763468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.763435    4531 generic.go:356] "Generic (PLEG): container finished" podID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerID="fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a" exitCode=60
Nov 01 23:00:05.763553 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.763532    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerDied","Data":"fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a"}
Nov 01 23:00:05.763843 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.763819    4531 scope.go:122] "RemoveContainer" containerID="fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a"
Nov 01 23:00:05.764071 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.764043    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm_odh-ai-gateway-infra(a2cccee6-d93f-4447-af1f-db2de26c124b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b"
Nov 01 23:00:05.765488 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.765467    4531 generic.go:356] "Generic (PLEG): container finished" podID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerID="feaa89ee8271532ec7ffca3b77e1c7ffc4a70144a2b0ff43e62300f62ded168d" exitCode=60
Nov 01 23:00:05.765569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.765547    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" event={"ID":"aec38feb-bb0d-49e6-bb55-5a5693ccca4d","Type":"ContainerDied","Data":"feaa89ee8271532ec7ffca3b77e1c7ffc4a70144a2b0ff43e62300f62ded168d"}
Nov 01 23:00:05.765738 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.765712    4531 scope.go:122] "RemoveContainer" containerID="feaa89ee8271532ec7ffca3b77e1c7ffc4a70144a2b0ff43e62300f62ded168d"
Nov 01 23:00:05.765962 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.765940    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j_odh-ai-gateway-infra(aec38feb-bb0d-49e6-bb55-5a5693ccca4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d"
Nov 01 23:00:05.767366 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.767347    4531 generic.go:356] "Generic (PLEG): container finished" podID="086dc50c-8e76-437f-af84-ca77b63d7228" containerID="d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a" exitCode=60
Nov 01 23:00:05.767448 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.767418    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerDied","Data":"d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a"}
Nov 01 23:00:05.767589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.767576    4531 scope.go:122] "RemoveContainer" containerID="d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a"
Nov 01 23:00:05.767824 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.767804    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26_odh-ai-gateway-infra(086dc50c-8e76-437f-af84-ca77b63d7228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" podUID="086dc50c-8e76-437f-af84-ca77b63d7228"
Nov 01 23:00:05.769181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.769161    4531 generic.go:356] "Generic (PLEG): container finished" podID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerID="5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919" exitCode=60
Nov 01 23:00:05.769264 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.769185    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" event={"ID":"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2","Type":"ContainerDied","Data":"5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919"}
Nov 01 23:00:05.769469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.769451    4531 scope.go:122] "RemoveContainer" containerID="5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919"
Nov 01 23:00:05.769793 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.769761    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd_odh-ai-gateway-infra(a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2"
Nov 01 23:00:05.770894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.770879    4531 scope.go:122] "RemoveContainer" containerID="6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d"
Nov 01 23:00:05.771118 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.771094    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2_odh-ai-gateway-infra(ec531e1a-fb93-4b56-afed-7a77461092cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd"
Nov 01 23:00:05.772573 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.772555    4531 generic.go:356] "Generic (PLEG): container finished" podID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerID="684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3" exitCode=60
Nov 01 23:00:05.772696 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.772679    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" event={"ID":"bc33a895-3e28-4ab8-9e42-7076c0f9380d","Type":"ContainerDied","Data":"684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3"}
Nov 01 23:00:05.773184 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.772867    4531 scope.go:122] "RemoveContainer" containerID="684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3"
Nov 01 23:00:05.773184 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:05.773130    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh_odh-ai-gateway-infra(bc33a895-3e28-4ab8-9e42-7076c0f9380d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d"
Nov 01 23:00:05.779072 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.779052    4531 scope.go:122] "RemoveContainer" containerID="5ab5e2b0691f2659de3575e2fe79e267e1a39f9083590b6e8d626ecbe366a4a5"
Nov 01 23:00:05.794091 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.794074    4531 scope.go:122] "RemoveContainer" containerID="cf445c8d33e2109d75ee500807033e383eee45a11cd3bccb13a96be3dca11017"
Nov 01 23:00:05.808949 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.808930    4531 scope.go:122] "RemoveContainer" containerID="e26b800bb25ebbb2ad6e24f574322cf069c6a3dfd2d13cb8cb3215310b3d6262"
Nov 01 23:00:05.822358 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.822339    4531 scope.go:122] "RemoveContainer" containerID="7b4b50aec381d6ec96b17ae41f3af0f018c2b01341878b45f0c3d5d0f8d0f7a7"
Nov 01 23:00:05.836416 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.836400    4531 scope.go:122] "RemoveContainer" containerID="3469d491ea1675f2b60ffbe8dc47a1c9e56b8cd8b2d1e1aa92a81eecb274c893"
Nov 01 23:00:05.851624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.851608    4531 scope.go:122] "RemoveContainer" containerID="6504bde4bcaf24b7dacb56e77faf32b13b70eab2018daa796235c84fe4b16fad"
Nov 01 23:00:05.864970 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.864954    4531 scope.go:122] "RemoveContainer" containerID="04c42daee3ef811a57d5fd4bddf022305fff2d1b5dc7ad6769fe9b3a65b947ea"
Nov 01 23:00:05.878510 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:05.878493    4531 scope.go:122] "RemoveContainer" containerID="1d62d17cdb42bf4d3dcdf0cf3ba50d42d1c3a2153beef737a2870ff8cc2b3f07"
Nov 01 23:00:06.079608 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.079517    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:06.793587 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.793555    4531 scope.go:122] "RemoveContainer" containerID="7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca"
Nov 01 23:00:06.794631 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.794598    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq_odh-ai-gateway-infra(99225305-c4f2-4987-9fcd-833f501913a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" podUID="99225305-c4f2-4987-9fcd-833f501913a8"
Nov 01 23:00:06.796207 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.796181    4531 scope.go:122] "RemoveContainer" containerID="f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109"
Nov 01 23:00:06.796608 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.796583    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p_odh-ai-gateway-infra(c727fe96-45a3-4928-b0b0-20a52737f04b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b"
Nov 01 23:00:06.802595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.802575    4531 scope.go:122] "RemoveContainer" containerID="fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a"
Nov 01 23:00:06.802874 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.802846    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm_odh-ai-gateway-infra(a2cccee6-d93f-4447-af1f-db2de26c124b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b"
Nov 01 23:00:06.804490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.804470    4531 scope.go:122] "RemoveContainer" containerID="feaa89ee8271532ec7ffca3b77e1c7ffc4a70144a2b0ff43e62300f62ded168d"
Nov 01 23:00:06.804761 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.804712    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j_odh-ai-gateway-infra(aec38feb-bb0d-49e6-bb55-5a5693ccca4d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d"
Nov 01 23:00:06.806150 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.806129    4531 scope.go:122] "RemoveContainer" containerID="d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a"
Nov 01 23:00:06.806367 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.806345    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26_odh-ai-gateway-infra(086dc50c-8e76-437f-af84-ca77b63d7228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" podUID="086dc50c-8e76-437f-af84-ca77b63d7228"
Nov 01 23:00:06.807629 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.807608    4531 scope.go:122] "RemoveContainer" containerID="5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919"
Nov 01 23:00:06.807862 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.807844    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd_odh-ai-gateway-infra(a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2"
Nov 01 23:00:06.809425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.809405    4531 scope.go:122] "RemoveContainer" containerID="684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3"
Nov 01 23:00:06.809618 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.809596    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh_odh-ai-gateway-infra(bc33a895-3e28-4ab8-9e42-7076c0f9380d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d"
Nov 01 23:00:06.810894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.810876    4531 scope.go:122] "RemoveContainer" containerID="a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c"
Nov 01 23:00:06.811107 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.811085    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx_odh-ai-gateway-infra(51b613b6-48e0-45ce-8cff-9daf0cecc43f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f"
Nov 01 23:00:06.812628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.812611    4531 scope.go:122] "RemoveContainer" containerID="d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4"
Nov 01 23:00:06.812904 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.812882    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf_odh-ai-gateway-infra(e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7"
Nov 01 23:00:06.814191 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:06.814174    4531 scope.go:122] "RemoveContainer" containerID="ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac"
Nov 01 23:00:06.814355 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:06.814339    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7_odh-ai-gateway-infra(980b4aa0-cf34-4e16-b97f-417bc731da72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72"
Nov 01 23:00:07.622735 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:07.622688    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:07.905227 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:07.905140    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:08.085006 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.084971    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"]
Nov 01 23:00:08.089980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.089951    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1f0206-rzvvl"]
Nov 01 23:00:08.187168 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.187093    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:08.452278 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.452242    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:08.617789 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.617751    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"]
Nov 01 23:00:08.618316 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.618280    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy" containerID="cri-o://68a755e374a55ff1f0e97cc71244c52ea909a6b592f2483f70b41fce3d9f24ed" gracePeriod=30
Nov 01 23:00:08.624001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.623977    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"]
Nov 01 23:00:08.624170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.624150    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy" containerID="cri-o://7fac0d27fcfa5dc9979c6deae183ccc07811135a5bc373fd01a849eacd96f7e2" gracePeriod=30
Nov 01 23:00:08.769803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.769594    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:08.868447 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:08.867929    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9987bf72-4ac2-471f-989b-6ec9b2e07376" path="/var/lib/kubelet/pods/9987bf72-4ac2-471f-989b-6ec9b2e07376/volumes"
Nov 01 23:00:09.005897 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:09.005862    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:09.659943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:09.659909    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"]
Nov 01 23:00:09.764935 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:09.764524    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"]
Nov 01 23:00:10.092868 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:10.092833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:10.299072 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:10.299022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:12.449421 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:12.449387    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:12.487940 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:12.487904    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"]
Nov 01 23:00:12.574567 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:12.574525    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:00:12.892548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:12.892515    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:12.907304 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:12.907277    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"
Nov 01 23:00:12.907469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:12.907312    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j" event={"ID":"aec38feb-bb0d-49e6-bb55-5a5693ccca4d","Type":"ContainerDied","Data":"be2fb5ca7ac7aeb0b4b76387cd6da6be1750316c5793dcdbcbc97c05ba720f3c"}
Nov 01 23:00:12.907469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:12.907384    4531 scope.go:122] "RemoveContainer" containerID="feaa89ee8271532ec7ffca3b77e1c7ffc4a70144a2b0ff43e62300f62ded168d"
Nov 01 23:00:13.060813 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.060778    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg\" (UniqueName: \"kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg\") pod \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") "
Nov 01 23:00:13.061026 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.061010    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca\") pod \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\" (UID: \"aec38feb-bb0d-49e6-bb55-5a5693ccca4d\") "
Nov 01 23:00:13.061420 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.061396    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca" pod "aec38feb-bb0d-49e6-bb55-5a5693ccca4d" (UID: "aec38feb-bb0d-49e6-bb55-5a5693ccca4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:13.061623 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.061609    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:13.062924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.062901    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg" pod "aec38feb-bb0d-49e6-bb55-5a5693ccca4d" (UID: "aec38feb-bb0d-49e6-bb55-5a5693ccca4d"). InnerVolumeSpecName "kube-api-access-4cjgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:13.162970 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.162901    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cjgg\" (UniqueName: \"kubernetes.io/projected/aec38feb-bb0d-49e6-bb55-5a5693ccca4d-kube-api-access-4cjgg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:13.339174 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.339131    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"]
Nov 01 23:00:13.344871 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.344841    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-f5b935-29892900-qtv2j"]
Nov 01 23:00:13.792638 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.792598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:13.982172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.980059    4531 generic.go:356] "Generic (PLEG): container finished" podID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerID="7fac0d27fcfa5dc9979c6deae183ccc07811135a5bc373fd01a849eacd96f7e2" exitCode=0
Nov 01 23:00:13.982172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.980182    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" event={"ID":"ce3e3eed-c404-4205-b21f-fc5d8c982229","Type":"ContainerDied","Data":"7fac0d27fcfa5dc9979c6deae183ccc07811135a5bc373fd01a849eacd96f7e2"}
Nov 01 23:00:13.985779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.984871    4531 generic.go:356] "Generic (PLEG): container finished" podID="06062fa0-7733-4503-b2b6-10065547bf34" containerID="68a755e374a55ff1f0e97cc71244c52ea909a6b592f2483f70b41fce3d9f24ed" exitCode=0
Nov 01 23:00:13.985779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:13.985131    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" event={"ID":"06062fa0-7733-4503-b2b6-10065547bf34","Type":"ContainerDied","Data":"68a755e374a55ff1f0e97cc71244c52ea909a6b592f2483f70b41fce3d9f24ed"}
Nov 01 23:00:14.111936 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.111878    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 23:00:14.133082 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.133043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:14.215491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215420    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215467    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215505    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215550    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215569    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215594    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215672    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.215769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.216086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.215826    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert\") pod \"ce3e3eed-c404-4205-b21f-fc5d8c982229\" (UID: \"ce3e3eed-c404-4205-b21f-fc5d8c982229\") "
Nov 01 23:00:14.216698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.216665    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.216869 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.216776    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.217394 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.217304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.217617 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.217585    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.217858 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.217830    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.219980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.219945    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "kube-api-access-nd49q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.220085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.220038    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.220085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.220039    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.224491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.223921    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy" pod "ce3e3eed-c404-4205-b21f-fc5d8c982229" (UID: "ce3e3eed-c404-4205-b21f-fc5d8c982229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.317805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.317775    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318021 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318006    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318135 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318121    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd49q\" (UniqueName: \"kubernetes.io/projected/ce3e3eed-c404-4205-b21f-fc5d8c982229-kube-api-access-nd49q\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318221 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318210    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318307 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318297    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318378    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318473    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318554    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3e3eed-c404-4205-b21f-fc5d8c982229-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.318644 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.318634    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3e3eed-c404-4205-b21f-fc5d8c982229-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.354978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.354944    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 23:00:14.420270 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420236    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420500 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420286    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420500 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420342    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420500 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420448    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420500 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420495    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420580    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420605    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.420768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420617    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420759    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420994 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420826    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo\") pod \"06062fa0-7733-4503-b2b6-10065547bf34\" (UID: \"06062fa0-7733-4503-b2b6-10065547bf34\") "
Nov 01 23:00:14.420994 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.420929    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.422690 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.422627    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.422861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.422696    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.424208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.423520    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "kube-api-access-824bh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.424208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.423850    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.424208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.424075    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:14.424208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.424098    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.424530 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.424387    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:14.425155 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.425098    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:14.425254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.425166    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo" pod "06062fa0-7733-4503-b2b6-10065547bf34" (UID: "06062fa0-7733-4503-b2b6-10065547bf34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:14.426471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.426438    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.523964    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.523990    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824bh\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-kube-api-access-824bh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.524001    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06062fa0-7733-4503-b2b6-10065547bf34-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.524018    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06062fa0-7733-4503-b2b6-10065547bf34-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.524032    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06062fa0-7733-4503-b2b6-10065547bf34-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.524046    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.537803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.524059    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06062fa0-7733-4503-b2b6-10065547bf34-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:14.621641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.621608    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"]
Nov 01 23:00:14.647407 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.647379    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-f5b935-m64sg"]
Nov 01 23:00:14.660407 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.660364    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5" podUID="2b8a2634-02cf-48b2-9679-037cb4a2783d" containerName="payload-processing" containerID="cri-o://5dc3ec3c71d4e86ff43dbb91e52661d2d0eae9bc9a71a8378b064bbfcdf900b1" gracePeriod=25
Nov 01 23:00:14.765717 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.765677    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj" podUID="1fa3111e-1565-4bcd-913a-c3161e31c140" containerName="payload-pre-processing" containerID="cri-o://17e1fbfccee397b1973740ee21c2de45efe8e60c340b807828ee8eddfa6a1393" gracePeriod=25
Nov 01 23:00:14.810620 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.809225    4531 scope.go:122] "RemoveContainer" containerID="50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4"
Nov 01 23:00:14.810620 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.809379    4531 scope.go:122] "RemoveContainer" containerID="6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d"
Nov 01 23:00:14.821606 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.821580    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="548be2ce-a087-45ed-b703-d2e322966045" path="/var/lib/kubelet/pods/548be2ce-a087-45ed-b703-d2e322966045/volumes"
Nov 01 23:00:14.822519 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:14.822144    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" path="/var/lib/kubelet/pods/aec38feb-bb0d-49e6-bb55-5a5693ccca4d/volumes"
Nov 01 23:00:15.007599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.007408    4531 generic.go:356] "Generic (PLEG): container finished" podID="1fa3111e-1565-4bcd-913a-c3161e31c140" containerID="17e1fbfccee397b1973740ee21c2de45efe8e60c340b807828ee8eddfa6a1393" exitCode=0
Nov 01 23:00:15.007599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.007528    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj" event={"ID":"1fa3111e-1565-4bcd-913a-c3161e31c140","Type":"ContainerDied","Data":"17e1fbfccee397b1973740ee21c2de45efe8e60c340b807828ee8eddfa6a1393"}
Nov 01 23:00:15.010594 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.010355    4531 generic.go:356] "Generic (PLEG): container finished" podID="2b8a2634-02cf-48b2-9679-037cb4a2783d" containerID="5dc3ec3c71d4e86ff43dbb91e52661d2d0eae9bc9a71a8378b064bbfcdf900b1" exitCode=0
Nov 01 23:00:15.010594 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.010531    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5" event={"ID":"2b8a2634-02cf-48b2-9679-037cb4a2783d","Type":"ContainerDied","Data":"5dc3ec3c71d4e86ff43dbb91e52661d2d0eae9bc9a71a8378b064bbfcdf900b1"}
Nov 01 23:00:15.014809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.012580    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv" event={"ID":"ce3e3eed-c404-4205-b21f-fc5d8c982229","Type":"ContainerDied","Data":"c8a59d4309895caa92faebde4efa7fa3ddec216c074b73e7605cea014d94b62a"}
Nov 01 23:00:15.014809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.012627    4531 scope.go:122] "RemoveContainer" containerID="7fac0d27fcfa5dc9979c6deae183ccc07811135a5bc373fd01a849eacd96f7e2"
Nov 01 23:00:15.014809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.012807    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"
Nov 01 23:00:15.022697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.022673    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784" event={"ID":"06062fa0-7733-4503-b2b6-10065547bf34","Type":"ContainerDied","Data":"d5ada9ec8369e65d50681f8c3294a7b9b7ebe4b13a6a6ef995635a133aa22e4c"}
Nov 01 23:00:15.022950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.022915    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"
Nov 01 23:00:15.083028 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.083003    4531 scope.go:122] "RemoveContainer" containerID="68a755e374a55ff1f0e97cc71244c52ea909a6b592f2483f70b41fce3d9f24ed"
Nov 01 23:00:15.121356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.121335    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 23:00:15.124121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.124102    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 23:00:15.131135 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.131114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw\" (UniqueName: \"kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw\") pod \"2b8a2634-02cf-48b2-9679-037cb4a2783d\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") "
Nov 01 23:00:15.131363 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.131345    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9\" (UniqueName: \"kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9\") pod \"1fa3111e-1565-4bcd-913a-c3161e31c140\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") "
Nov 01 23:00:15.131409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.131397    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume\") pod \"2b8a2634-02cf-48b2-9679-037cb4a2783d\" (UID: \"2b8a2634-02cf-48b2-9679-037cb4a2783d\") "
Nov 01 23:00:15.131452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.131434    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume\") pod \"1fa3111e-1565-4bcd-913a-c3161e31c140\" (UID: \"1fa3111e-1565-4bcd-913a-c3161e31c140\") "
Nov 01 23:00:15.131839 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.131812    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume" pod "2b8a2634-02cf-48b2-9679-037cb4a2783d" (UID: "2b8a2634-02cf-48b2-9679-037cb4a2783d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:15.131985 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.131879    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume" pod "1fa3111e-1565-4bcd-913a-c3161e31c140" (UID: "1fa3111e-1565-4bcd-913a-c3161e31c140"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:15.132155 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.132136    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fa3111e-1565-4bcd-913a-c3161e31c140-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:15.132235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.132162    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b8a2634-02cf-48b2-9679-037cb4a2783d-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:15.133259 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.133236    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw" pod "2b8a2634-02cf-48b2-9679-037cb4a2783d" (UID: "2b8a2634-02cf-48b2-9679-037cb4a2783d"). InnerVolumeSpecName "kube-api-access-l9llw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:15.133416 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.133397    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9" pod "1fa3111e-1565-4bcd-913a-c3161e31c140" (UID: "1fa3111e-1565-4bcd-913a-c3161e31c140"). InnerVolumeSpecName "kube-api-access-nj8h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:15.145755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.145714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:15.189315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.189289    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"]
Nov 01 23:00:15.193987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.193965    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-8w784"]
Nov 01 23:00:15.233590 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.233570    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj8h9\" (UniqueName: \"kubernetes.io/projected/1fa3111e-1565-4bcd-913a-c3161e31c140-kube-api-access-nj8h9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:15.233590 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.233591    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9llw\" (UniqueName: \"kubernetes.io/projected/2b8a2634-02cf-48b2-9679-037cb4a2783d-kube-api-access-l9llw\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:15.311884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.311805    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"]
Nov 01 23:00:15.316958 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.316928    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-945efd02-openshift-default-974cc8b4f-fs6pv"]
Nov 01 23:00:15.421545 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.421514    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:15.689217 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.689143    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:15.809707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.809676    4531 scope.go:122] "RemoveContainer" containerID="7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca"
Nov 01 23:00:15.809910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.809877    4531 scope.go:122] "RemoveContainer" containerID="d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a"
Nov 01 23:00:15.968460 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.968406    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"]
Nov 01 23:00:15.977644 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.977620    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d68bab-wj7mz"]
Nov 01 23:00:15.999255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:15.999227    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:16.035354 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.035332    4531 generic.go:356] "Generic (PLEG): container finished" podID="39b0859f-d187-4896-9e50-f9d981010d7d" containerID="96e105fd8236dacdd27c134b782572a6ac998516bfbcf48bdfc69c611adfa05e" exitCode=60
Nov 01 23:00:16.035468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.035402    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" event={"ID":"39b0859f-d187-4896-9e50-f9d981010d7d","Type":"ContainerDied","Data":"96e105fd8236dacdd27c134b782572a6ac998516bfbcf48bdfc69c611adfa05e"}
Nov 01 23:00:16.035468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.035460    4531 scope.go:122] "RemoveContainer" containerID="50a100313bbf600cdaadcd362591c8009ecc543e231ff912706451f3c895c6f4"
Nov 01 23:00:16.035878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.035855    4531 scope.go:122] "RemoveContainer" containerID="96e105fd8236dacdd27c134b782572a6ac998516bfbcf48bdfc69c611adfa05e"
Nov 01 23:00:16.036216 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:16.036196    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj_odh-ai-gateway-infra(39b0859f-d187-4896-9e50-f9d981010d7d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" podUID="39b0859f-d187-4896-9e50-f9d981010d7d"
Nov 01 23:00:16.037003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.036987    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"
Nov 01 23:00:16.037090 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.037025    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj" event={"ID":"1fa3111e-1565-4bcd-913a-c3161e31c140","Type":"ContainerDied","Data":"08d735db855ebe5e107d6e7dd05b2c9c093cda3bb6632f712489ffb37af9c935"}
Nov 01 23:00:16.038461 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.038442    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"
Nov 01 23:00:16.038564 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.038466    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5" event={"ID":"2b8a2634-02cf-48b2-9679-037cb4a2783d","Type":"ContainerDied","Data":"96bbbfec8f78b3abc29da28269036e9b7429d83c19c5cf78221973734e707081"}
Nov 01 23:00:16.040470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.040451    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerStarted","Data":"da098436eeedc14e068ae9780aff67b698ad9e92476d29e405a3f9899582cdcd"}
Nov 01 23:00:16.097237 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.096906    4531 scope.go:122] "RemoveContainer" containerID="17e1fbfccee397b1973740ee21c2de45efe8e60c340b807828ee8eddfa6a1393"
Nov 01 23:00:16.121259 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.121236    4531 scope.go:122] "RemoveContainer" containerID="5dc3ec3c71d4e86ff43dbb91e52661d2d0eae9bc9a71a8378b064bbfcdf900b1"
Nov 01 23:00:16.223698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.223632    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:16.336363 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.333826    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"]
Nov 01 23:00:16.371890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.371803    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d68bab-547d986d59-bzktj"]
Nov 01 23:00:16.387053 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.386792    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"]
Nov 01 23:00:16.393570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.393543    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d68bab-755558dfdf-vx6p5"]
Nov 01 23:00:16.413445 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.413409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:16.809177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.809136    4531 scope.go:122] "RemoveContainer" containerID="d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4"
Nov 01 23:00:16.809435 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.809413    4531 scope.go:122] "RemoveContainer" containerID="fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a"
Nov 01 23:00:16.809524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.809457    4531 scope.go:122] "RemoveContainer" containerID="ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac"
Nov 01 23:00:16.827901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.827874    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06062fa0-7733-4503-b2b6-10065547bf34" path="/var/lib/kubelet/pods/06062fa0-7733-4503-b2b6-10065547bf34/volumes"
Nov 01 23:00:16.828434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.828417    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa3111e-1565-4bcd-913a-c3161e31c140" path="/var/lib/kubelet/pods/1fa3111e-1565-4bcd-913a-c3161e31c140/volumes"
Nov 01 23:00:16.828984 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.828967    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b8a2634-02cf-48b2-9679-037cb4a2783d" path="/var/lib/kubelet/pods/2b8a2634-02cf-48b2-9679-037cb4a2783d/volumes"
Nov 01 23:00:16.829975 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.829955    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86326a5d-e4f1-4038-9a92-453fa427443a" path="/var/lib/kubelet/pods/86326a5d-e4f1-4038-9a92-453fa427443a/volumes"
Nov 01 23:00:16.830455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:16.830440    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" path="/var/lib/kubelet/pods/ce3e3eed-c404-4205-b21f-fc5d8c982229/volumes"
Nov 01 23:00:17.052022 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.051995    4531 generic.go:356] "Generic (PLEG): container finished" podID="086dc50c-8e76-437f-af84-ca77b63d7228" containerID="fcb2fedfc11342cb357468525be43a22bba7bff7c1b0d12c422dadd68fa8fd06" exitCode=60
Nov 01 23:00:17.052473 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.052070    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerDied","Data":"fcb2fedfc11342cb357468525be43a22bba7bff7c1b0d12c422dadd68fa8fd06"}
Nov 01 23:00:17.052473 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.052137    4531 scope.go:122] "RemoveContainer" containerID="d22524b02f324440a438ebf9817cfd9ac7b2640ba260986b0054055621ab218a"
Nov 01 23:00:17.052582 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.052509    4531 scope.go:122] "RemoveContainer" containerID="fcb2fedfc11342cb357468525be43a22bba7bff7c1b0d12c422dadd68fa8fd06"
Nov 01 23:00:17.052889 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:17.052860    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26_odh-ai-gateway-infra(086dc50c-8e76-437f-af84-ca77b63d7228)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" podUID="086dc50c-8e76-437f-af84-ca77b63d7228"
Nov 01 23:00:17.055196 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.055177    4531 generic.go:356] "Generic (PLEG): container finished" podID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerID="da098436eeedc14e068ae9780aff67b698ad9e92476d29e405a3f9899582cdcd" exitCode=60
Nov 01 23:00:17.055274 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.055235    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerDied","Data":"da098436eeedc14e068ae9780aff67b698ad9e92476d29e405a3f9899582cdcd"}
Nov 01 23:00:17.055579 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.055565    4531 scope.go:122] "RemoveContainer" containerID="da098436eeedc14e068ae9780aff67b698ad9e92476d29e405a3f9899582cdcd"
Nov 01 23:00:17.055921 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:17.055898    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2_odh-ai-gateway-infra(ec531e1a-fb93-4b56-afed-7a77461092cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd"
Nov 01 23:00:17.058952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.058933    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerStarted","Data":"77e8046a44b268286901e3397eda02ce9b215c5c53091b8b66d9987fceea7a71"}
Nov 01 23:00:17.162157 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.162119    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"]
Nov 01 23:00:17.208361 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.208339    4531 scope.go:122] "RemoveContainer" containerID="6458ec893ffcc704da9ae89e69074410dde7b892ac9a38611863dcc449151d7d"
Nov 01 23:00:17.371706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.371672    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:17.472563 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.472528    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7\" (UniqueName: \"kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7\") pod \"39b0859f-d187-4896-9e50-f9d981010d7d\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") "
Nov 01 23:00:17.472843 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.472823    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca\") pod \"39b0859f-d187-4896-9e50-f9d981010d7d\" (UID: \"39b0859f-d187-4896-9e50-f9d981010d7d\") "
Nov 01 23:00:17.473183 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.473161    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca" pod "39b0859f-d187-4896-9e50-f9d981010d7d" (UID: "39b0859f-d187-4896-9e50-f9d981010d7d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:17.473379 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.473364    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/39b0859f-d187-4896-9e50-f9d981010d7d-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:17.474612 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.474576    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7" pod "39b0859f-d187-4896-9e50-f9d981010d7d" (UID: "39b0859f-d187-4896-9e50-f9d981010d7d"). InnerVolumeSpecName "kube-api-access-w8tm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:17.493508 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.493478    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"]
Nov 01 23:00:17.575270 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.575193    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8tm7\" (UniqueName: \"kubernetes.io/projected/39b0859f-d187-4896-9e50-f9d981010d7d-kube-api-access-w8tm7\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:17.809741 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.809705    4531 scope.go:122] "RemoveContainer" containerID="5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919"
Nov 01 23:00:17.809942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.809889    4531 scope.go:122] "RemoveContainer" containerID="a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c"
Nov 01 23:00:17.888960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:17.888926    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:18.079988 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.079933    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerStarted","Data":"6bb02b3f008d3effd50eb023dd7f006587aa8a2a30ebed4ebd26b86bd83dfcd3"}
Nov 01 23:00:18.080511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.080295    4531 scope.go:122] "RemoveContainer" containerID="6bb02b3f008d3effd50eb023dd7f006587aa8a2a30ebed4ebd26b86bd83dfcd3"
Nov 01 23:00:18.080751 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:18.080604    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf_odh-ai-gateway-infra(e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7"
Nov 01 23:00:18.083638 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.083585    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerStarted","Data":"c9b10ab32338e9a9b9ac7b733eda055cc0295cd3bd52fbee4a7dc2455a705963"}
Nov 01 23:00:18.084065 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.084020    4531 scope.go:122] "RemoveContainer" containerID="c9b10ab32338e9a9b9ac7b733eda055cc0295cd3bd52fbee4a7dc2455a705963"
Nov 01 23:00:18.084333 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:18.084309    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7_odh-ai-gateway-infra(980b4aa0-cf34-4e16-b97f-417bc731da72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72"
Nov 01 23:00:18.085490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.085469    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"
Nov 01 23:00:18.085638 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.085509    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj" event={"ID":"39b0859f-d187-4896-9e50-f9d981010d7d","Type":"ContainerDied","Data":"e18a0ee605507d07a9d977fab4e8f1b1be019530d095b0311c5253a239fadbf7"}
Nov 01 23:00:18.085638 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.085562    4531 scope.go:122] "RemoveContainer" containerID="96e105fd8236dacdd27c134b782572a6ac998516bfbcf48bdfc69c611adfa05e"
Nov 01 23:00:18.088844 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.088314    4531 generic.go:356] "Generic (PLEG): container finished" podID="99225305-c4f2-4987-9fcd-833f501913a8" containerID="77e8046a44b268286901e3397eda02ce9b215c5c53091b8b66d9987fceea7a71" exitCode=60
Nov 01 23:00:18.088844 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.088435    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerDied","Data":"77e8046a44b268286901e3397eda02ce9b215c5c53091b8b66d9987fceea7a71"}
Nov 01 23:00:18.088844 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.088805    4531 scope.go:122] "RemoveContainer" containerID="77e8046a44b268286901e3397eda02ce9b215c5c53091b8b66d9987fceea7a71"
Nov 01 23:00:18.089411 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:18.089343    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq_odh-ai-gateway-infra(99225305-c4f2-4987-9fcd-833f501913a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" podUID="99225305-c4f2-4987-9fcd-833f501913a8"
Nov 01 23:00:18.092016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.091994    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerStarted","Data":"9dd5b635b945c24dff9d971c16b0a64f0d90e6296e1272267e7af3bafbad8bca"}
Nov 01 23:00:18.092362 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.092345    4531 scope.go:122] "RemoveContainer" containerID="9dd5b635b945c24dff9d971c16b0a64f0d90e6296e1272267e7af3bafbad8bca"
Nov 01 23:00:18.092615 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:18.092595    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm_odh-ai-gateway-infra(a2cccee6-d93f-4447-af1f-db2de26c124b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b"
Nov 01 23:00:18.211875 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.211850    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"]
Nov 01 23:00:18.281137 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.281114    4531 scope.go:122] "RemoveContainer" containerID="7edd8a30743a24daac6cb8bfa05ad8ceb4a86c29cbc41cca40a0b0a1a0b20aca"
Nov 01 23:00:18.300529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.300288    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:18.387800 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.387769    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca\") pod \"ec531e1a-fb93-4b56-afed-7a77461092cd\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") "
Nov 01 23:00:18.387986 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.387959    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc\" (UniqueName: \"kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc\") pod \"ec531e1a-fb93-4b56-afed-7a77461092cd\" (UID: \"ec531e1a-fb93-4b56-afed-7a77461092cd\") "
Nov 01 23:00:18.389177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.389152    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca" pod "ec531e1a-fb93-4b56-afed-7a77461092cd" (UID: "ec531e1a-fb93-4b56-afed-7a77461092cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:18.391545 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.391503    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc" pod "ec531e1a-fb93-4b56-afed-7a77461092cd" (UID: "ec531e1a-fb93-4b56-afed-7a77461092cd"). InnerVolumeSpecName "kube-api-access-f87dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:18.418001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.417980    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:18.489829 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.489806    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca\") pod \"086dc50c-8e76-437f-af84-ca77b63d7228\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") "
Nov 01 23:00:18.489977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.489875    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t\" (UniqueName: \"kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t\") pod \"086dc50c-8e76-437f-af84-ca77b63d7228\" (UID: \"086dc50c-8e76-437f-af84-ca77b63d7228\") "
Nov 01 23:00:18.490124 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.490102    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca" pod "086dc50c-8e76-437f-af84-ca77b63d7228" (UID: "086dc50c-8e76-437f-af84-ca77b63d7228"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:18.490555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.490535    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f87dc\" (UniqueName: \"kubernetes.io/projected/ec531e1a-fb93-4b56-afed-7a77461092cd-kube-api-access-f87dc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:18.490555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.490555    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/086dc50c-8e76-437f-af84-ca77b63d7228-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:18.490744 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.490565    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec531e1a-fb93-4b56-afed-7a77461092cd-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:18.492086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.492065    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t" pod "086dc50c-8e76-437f-af84-ca77b63d7228" (UID: "086dc50c-8e76-437f-af84-ca77b63d7228"). InnerVolumeSpecName "kube-api-access-gf85t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:18.527377 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.527347    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"]
Nov 01 23:00:18.591389 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.591359    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf85t\" (UniqueName: \"kubernetes.io/projected/086dc50c-8e76-437f-af84-ca77b63d7228-kube-api-access-gf85t\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:18.809580 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.809551    4531 scope.go:122] "RemoveContainer" containerID="f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109"
Nov 01 23:00:18.809744 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:18.809706    4531 scope.go:122] "RemoveContainer" containerID="684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3"
Nov 01 23:00:19.071505 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.071460    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"]
Nov 01 23:00:19.078715 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.078687    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-103369-29892900-pkfdj"]
Nov 01 23:00:19.108999 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.108970    4531 generic.go:356] "Generic (PLEG): container finished" podID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerID="3e70aac77822127845dd22c7619e39fd3f8f5856fd295354784bf0b9e23bab43" exitCode=60
Nov 01 23:00:19.109343 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.108995    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerDied","Data":"3e70aac77822127845dd22c7619e39fd3f8f5856fd295354784bf0b9e23bab43"}
Nov 01 23:00:19.109343 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.109053    4531 scope.go:122] "RemoveContainer" containerID="a3ef49e8b449b6a0cfe802721ab3641db1c11e4e72b99d831422db2369d7d41c"
Nov 01 23:00:19.109692 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.109458    4531 scope.go:122] "RemoveContainer" containerID="3e70aac77822127845dd22c7619e39fd3f8f5856fd295354784bf0b9e23bab43"
Nov 01 23:00:19.109926 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:19.109900    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx_odh-ai-gateway-infra(51b613b6-48e0-45ce-8cff-9daf0cecc43f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f"
Nov 01 23:00:19.111571 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.111550    4531 generic.go:356] "Generic (PLEG): container finished" podID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerID="6bb02b3f008d3effd50eb023dd7f006587aa8a2a30ebed4ebd26b86bd83dfcd3" exitCode=60
Nov 01 23:00:19.111665 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.111628    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerDied","Data":"6bb02b3f008d3effd50eb023dd7f006587aa8a2a30ebed4ebd26b86bd83dfcd3"}
Nov 01 23:00:19.112015 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.112000    4531 scope.go:122] "RemoveContainer" containerID="6bb02b3f008d3effd50eb023dd7f006587aa8a2a30ebed4ebd26b86bd83dfcd3"
Nov 01 23:00:19.112238 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:19.112220    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf_odh-ai-gateway-infra(e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7"
Nov 01 23:00:19.115509 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.115489    4531 generic.go:356] "Generic (PLEG): container finished" podID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerID="c9b10ab32338e9a9b9ac7b733eda055cc0295cd3bd52fbee4a7dc2455a705963" exitCode=60
Nov 01 23:00:19.115595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.115562    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerDied","Data":"c9b10ab32338e9a9b9ac7b733eda055cc0295cd3bd52fbee4a7dc2455a705963"}
Nov 01 23:00:19.115939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.115919    4531 scope.go:122] "RemoveContainer" containerID="c9b10ab32338e9a9b9ac7b733eda055cc0295cd3bd52fbee4a7dc2455a705963"
Nov 01 23:00:19.116162 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:19.116142    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7_odh-ai-gateway-infra(980b4aa0-cf34-4e16-b97f-417bc731da72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72"
Nov 01 23:00:19.120466 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.120442    4531 generic.go:356] "Generic (PLEG): container finished" podID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerID="9dd5b635b945c24dff9d971c16b0a64f0d90e6296e1272267e7af3bafbad8bca" exitCode=60
Nov 01 23:00:19.120538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.120490    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerDied","Data":"9dd5b635b945c24dff9d971c16b0a64f0d90e6296e1272267e7af3bafbad8bca"}
Nov 01 23:00:19.121051 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.121033    4531 scope.go:122] "RemoveContainer" containerID="9dd5b635b945c24dff9d971c16b0a64f0d90e6296e1272267e7af3bafbad8bca"
Nov 01 23:00:19.121350 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:19.121321    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm_odh-ai-gateway-infra(a2cccee6-d93f-4447-af1f-db2de26c124b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b"
Nov 01 23:00:19.122106 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.122077    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26" event={"ID":"086dc50c-8e76-437f-af84-ca77b63d7228","Type":"ContainerDied","Data":"119e5efe5a645d467a2605f5d3f4c8e278aaebabda40be54e93b497f7602a220"}
Nov 01 23:00:19.122255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.122088    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"
Nov 01 23:00:19.123987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.123969    4531 generic.go:356] "Generic (PLEG): container finished" podID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerID="4daf72eceb8df16bb21c4ae6415df1b0343938ee63b56b80a74790cf8b46f1ca" exitCode=60
Nov 01 23:00:19.124066 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.124040    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" event={"ID":"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2","Type":"ContainerDied","Data":"4daf72eceb8df16bb21c4ae6415df1b0343938ee63b56b80a74790cf8b46f1ca"}
Nov 01 23:00:19.124414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.124383    4531 scope.go:122] "RemoveContainer" containerID="4daf72eceb8df16bb21c4ae6415df1b0343938ee63b56b80a74790cf8b46f1ca"
Nov 01 23:00:19.124644 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:19.124623    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd_odh-ai-gateway-infra(a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2"
Nov 01 23:00:19.125454 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.125429    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2" event={"ID":"ec531e1a-fb93-4b56-afed-7a77461092cd","Type":"ContainerDied","Data":"cb3a79843e1be2e5a393b7e334406f3dd09f27e672b916225b9d39978807e34b"}
Nov 01 23:00:19.125568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.125441    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"
Nov 01 23:00:19.231416 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.231390    4531 scope.go:122] "RemoveContainer" containerID="d16201a0aef6a9538886d0d28b8c0774d8df5460282d665c2948dc066b4edba4"
Nov 01 23:00:19.249120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.249100    4531 scope.go:122] "RemoveContainer" containerID="ee6528281956e04ba4ff9676f30a0ba69407e8c8c3dac861a6f27286541079ac"
Nov 01 23:00:19.265898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.265704    4531 scope.go:122] "RemoveContainer" containerID="fc5b767f79c0c8db7a561e7a2ca306b735f5a42412a784e767629e645ae4d19a"
Nov 01 23:00:19.303938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.303916    4531 scope.go:122] "RemoveContainer" containerID="fcb2fedfc11342cb357468525be43a22bba7bff7c1b0d12c422dadd68fa8fd06"
Nov 01 23:00:19.305112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.305088    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:19.315234 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.315208    4531 scope.go:122] "RemoveContainer" containerID="5e9ad3fe30908bf2ecc1d362326515b602eb5cea7f3dba5d2321fd5d3c4a5919"
Nov 01 23:00:19.329230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.329213    4531 scope.go:122] "RemoveContainer" containerID="da098436eeedc14e068ae9780aff67b698ad9e92476d29e405a3f9899582cdcd"
Nov 01 23:00:19.413073 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.411099    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9\" (UniqueName: \"kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9\") pod \"99225305-c4f2-4987-9fcd-833f501913a8\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") "
Nov 01 23:00:19.413073 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.411343    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca\") pod \"99225305-c4f2-4987-9fcd-833f501913a8\" (UID: \"99225305-c4f2-4987-9fcd-833f501913a8\") "
Nov 01 23:00:19.413073 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.412249    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca" pod "99225305-c4f2-4987-9fcd-833f501913a8" (UID: "99225305-c4f2-4987-9fcd-833f501913a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:19.415545 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.415523    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9" pod "99225305-c4f2-4987-9fcd-833f501913a8" (UID: "99225305-c4f2-4987-9fcd-833f501913a8"). InnerVolumeSpecName "kube-api-access-fd2b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:19.443987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.443954    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"]
Nov 01 23:00:19.512704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.512682    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd2b9\" (UniqueName: \"kubernetes.io/projected/99225305-c4f2-4987-9fcd-833f501913a8-kube-api-access-fd2b9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:19.512704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.512706    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/99225305-c4f2-4987-9fcd-833f501913a8-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:19.558413 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.558386    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"]
Nov 01 23:00:19.735404 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:19.735370    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"]
Nov 01 23:00:20.116344 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.116273    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"]
Nov 01 23:00:20.121398 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.121377    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-a97162-29892900-mnz26"]
Nov 01 23:00:20.137229 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.137205    4531 generic.go:356] "Generic (PLEG): container finished" podID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerID="c5326292f5a1e4a4b1887eee1f79317c9f7aa749415667d0b415e1f6a7a8a096" exitCode=60
Nov 01 23:00:20.137334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.137277    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" event={"ID":"bc33a895-3e28-4ab8-9e42-7076c0f9380d","Type":"ContainerDied","Data":"c5326292f5a1e4a4b1887eee1f79317c9f7aa749415667d0b415e1f6a7a8a096"}
Nov 01 23:00:20.137416 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.137345    4531 scope.go:122] "RemoveContainer" containerID="684f94a387015412f7ef198f855c5f8233ccee46d030e07a2ba5719dd2f639b3"
Nov 01 23:00:20.137758 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.137722    4531 scope.go:122] "RemoveContainer" containerID="c5326292f5a1e4a4b1887eee1f79317c9f7aa749415667d0b415e1f6a7a8a096"
Nov 01 23:00:20.138083 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:20.138057    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh_odh-ai-gateway-infra(bc33a895-3e28-4ab8-9e42-7076c0f9380d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d"
Nov 01 23:00:20.144711 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.144691    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"
Nov 01 23:00:20.144832 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.144695    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq" event={"ID":"99225305-c4f2-4987-9fcd-833f501913a8","Type":"ContainerDied","Data":"e4b314dcfdd0bbeca29676825f4bd8787bdd6877f570d64510d59204064d6655"}
Nov 01 23:00:20.147345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.147318    4531 generic.go:356] "Generic (PLEG): container finished" podID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerID="35f6012da2702e6e7d1c18a4b5f05d810b362bb8a3efa483f52677a5ec144ead" exitCode=60
Nov 01 23:00:20.147450 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.147346    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerDied","Data":"35f6012da2702e6e7d1c18a4b5f05d810b362bb8a3efa483f52677a5ec144ead"}
Nov 01 23:00:20.148839 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.148807    4531 scope.go:122] "RemoveContainer" containerID="35f6012da2702e6e7d1c18a4b5f05d810b362bb8a3efa483f52677a5ec144ead"
Nov 01 23:00:20.149177 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:20.149151    4531 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p_odh-ai-gateway-infra(c727fe96-45a3-4928-b0b0-20a52737f04b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b"
Nov 01 23:00:20.162863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.162604    4531 scope.go:122] "RemoveContainer" containerID="77e8046a44b268286901e3397eda02ce9b215c5c53091b8b66d9987fceea7a71"
Nov 01 23:00:20.175566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.175536    4531 scope.go:122] "RemoveContainer" containerID="f774eb842e20213653840ddda42a15d73902a4d7370084c669b3a13c1b558109"
Nov 01 23:00:20.249348 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.249101    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"]
Nov 01 23:00:20.285806 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.285777    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"]
Nov 01 23:00:20.298131 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.298088    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-2b0866-29892900-mqlz2"]
Nov 01 23:00:20.387758 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.386111    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:20.397923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.397898    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:20.424282 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.424256    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca\") pod \"980b4aa0-cf34-4e16-b97f-417bc731da72\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") "
Nov 01 23:00:20.424405 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.424388    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t\" (UniqueName: \"kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t\") pod \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") "
Nov 01 23:00:20.424522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.424506    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca\") pod \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\" (UID: \"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7\") "
Nov 01 23:00:20.424905 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.424863    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca" pod "e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" (UID: "e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:20.425109 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.425090    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq\" (UniqueName: \"kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq\") pod \"980b4aa0-cf34-4e16-b97f-417bc731da72\" (UID: \"980b4aa0-cf34-4e16-b97f-417bc731da72\") "
Nov 01 23:00:20.426316 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.426293    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.428063 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.428024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t" pod "e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" (UID: "e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7"). InnerVolumeSpecName "kube-api-access-zvh8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:20.428480 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.428249    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca" pod "980b4aa0-cf34-4e16-b97f-417bc731da72" (UID: "980b4aa0-cf34-4e16-b97f-417bc731da72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:20.431546 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.431519    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq" pod "980b4aa0-cf34-4e16-b97f-417bc731da72" (UID: "980b4aa0-cf34-4e16-b97f-417bc731da72"). InnerVolumeSpecName "kube-api-access-x9mzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:20.446865 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.446753    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"]
Nov 01 23:00:20.461381 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.461302    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:20.532296 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.532181    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca\") pod \"a2cccee6-d93f-4447-af1f-db2de26c124b\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") "
Nov 01 23:00:20.532296 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.532214    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s\" (UniqueName: \"kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s\") pod \"a2cccee6-d93f-4447-af1f-db2de26c124b\" (UID: \"a2cccee6-d93f-4447-af1f-db2de26c124b\") "
Nov 01 23:00:20.539864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.532559    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca" pod "a2cccee6-d93f-4447-af1f-db2de26c124b" (UID: "a2cccee6-d93f-4447-af1f-db2de26c124b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:20.539864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.533074    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvh8t\" (UniqueName: \"kubernetes.io/projected/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7-kube-api-access-zvh8t\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.539864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.533100    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2cccee6-d93f-4447-af1f-db2de26c124b-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.539864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.533116    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9mzq\" (UniqueName: \"kubernetes.io/projected/980b4aa0-cf34-4e16-b97f-417bc731da72-kube-api-access-x9mzq\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.539864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.533131    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/980b4aa0-cf34-4e16-b97f-417bc731da72-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.539864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.538911    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s" pod "a2cccee6-d93f-4447-af1f-db2de26c124b" (UID: "a2cccee6-d93f-4447-af1f-db2de26c124b"). InnerVolumeSpecName "kube-api-access-nhs4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:20.545167 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.545139    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:20.634242 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.634160    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca\") pod \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") "
Nov 01 23:00:20.634397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.634378    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg\" (UniqueName: \"kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg\") pod \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\" (UID: \"51b613b6-48e0-45ce-8cff-9daf0cecc43f\") "
Nov 01 23:00:20.634879 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.634850    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca" pod "51b613b6-48e0-45ce-8cff-9daf0cecc43f" (UID: "51b613b6-48e0-45ce-8cff-9daf0cecc43f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:20.635069 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.634899    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhs4s\" (UniqueName: \"kubernetes.io/projected/a2cccee6-d93f-4447-af1f-db2de26c124b-kube-api-access-nhs4s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.637323 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.637280    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg" pod "51b613b6-48e0-45ce-8cff-9daf0cecc43f" (UID: "51b613b6-48e0-45ce-8cff-9daf0cecc43f"). InnerVolumeSpecName "kube-api-access-chbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:20.661717 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.661697    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:20.707335 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.707279    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"]
Nov 01 23:00:20.711918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.711895    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-bd7ff8-29892900-946dq"]
Nov 01 23:00:20.736334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.736311    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt\" (UniqueName: \"kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt\") pod \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") "
Nov 01 23:00:20.736456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.736437    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca\") pod \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\" (UID: \"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2\") "
Nov 01 23:00:20.736803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.736780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca" pod "a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" (UID: "a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:20.737019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.737003    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51b613b6-48e0-45ce-8cff-9daf0cecc43f-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.737088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.737021    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.737088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.737031    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chbwg\" (UniqueName: \"kubernetes.io/projected/51b613b6-48e0-45ce-8cff-9daf0cecc43f-kube-api-access-chbwg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:20.739084 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.739057    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt" pod "a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" (UID: "a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2"). InnerVolumeSpecName "kube-api-access-spgrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:20.819530 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.819498    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" path="/var/lib/kubelet/pods/086dc50c-8e76-437f-af84-ca77b63d7228/volumes"
Nov 01 23:00:20.819987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.819967    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" path="/var/lib/kubelet/pods/39b0859f-d187-4896-9e50-f9d981010d7d/volumes"
Nov 01 23:00:20.820424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.820408    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99225305-c4f2-4987-9fcd-833f501913a8" path="/var/lib/kubelet/pods/99225305-c4f2-4987-9fcd-833f501913a8/volumes"
Nov 01 23:00:20.821140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.821126    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" path="/var/lib/kubelet/pods/ec531e1a-fb93-4b56-afed-7a77461092cd/volumes"
Nov 01 23:00:20.837825 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:20.837799    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spgrt\" (UniqueName: \"kubernetes.io/projected/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2-kube-api-access-spgrt\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.165948 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.165893    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:21.173833 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.173805    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"
Nov 01 23:00:21.174469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.174442    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7" event={"ID":"980b4aa0-cf34-4e16-b97f-417bc731da72","Type":"ContainerDied","Data":"19f50cace0bd9aa115dccef5c2165c244a14c8080103dd0f1e165582cd70e391"}
Nov 01 23:00:21.174620 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.174502    4531 scope.go:122] "RemoveContainer" containerID="c9b10ab32338e9a9b9ac7b733eda055cc0295cd3bd52fbee4a7dc2455a705963"
Nov 01 23:00:21.182367 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.181382    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm" event={"ID":"a2cccee6-d93f-4447-af1f-db2de26c124b","Type":"ContainerDied","Data":"50e5e923673e8901b55bb973a651318dea9eaa142237649b8d5028d66e3773ef"}
Nov 01 23:00:21.182367 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.181476    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"
Nov 01 23:00:21.193624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.191456    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd" event={"ID":"a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2","Type":"ContainerDied","Data":"3f00bd98c989a2af39dd8a427e13b665e0f49a510df21963a1df88c9f7dacfca"}
Nov 01 23:00:21.193624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.191588    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"
Nov 01 23:00:21.201138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.201111    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx" event={"ID":"51b613b6-48e0-45ce-8cff-9daf0cecc43f","Type":"ContainerDied","Data":"bf72f045e7edc5ed9640ea2590de1089b1d9325ec878da32dc8f90bc93f04a44"}
Nov 01 23:00:21.201409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.201393    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"
Nov 01 23:00:21.206687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.206666    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf" event={"ID":"e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7","Type":"ContainerDied","Data":"2d08f49a3781e4635556fd0d4f5569240dd971cb1970fb17b3aa611856efa3ed"}
Nov 01 23:00:21.206967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.206943    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"
Nov 01 23:00:21.215842 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.211902    4531 scope.go:122] "RemoveContainer" containerID="9dd5b635b945c24dff9d971c16b0a64f0d90e6296e1272267e7af3bafbad8bca"
Nov 01 23:00:21.226887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.226869    4531 scope.go:122] "RemoveContainer" containerID="4daf72eceb8df16bb21c4ae6415df1b0343938ee63b56b80a74790cf8b46f1ca"
Nov 01 23:00:21.242112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.242089    4531 scope.go:122] "RemoveContainer" containerID="3e70aac77822127845dd22c7619e39fd3f8f5856fd295354784bf0b9e23bab43"
Nov 01 23:00:21.253711 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.253615    4531 scope.go:122] "RemoveContainer" containerID="6bb02b3f008d3effd50eb023dd7f006587aa8a2a30ebed4ebd26b86bd83dfcd3"
Nov 01 23:00:21.507052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.506762    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"]
Nov 01 23:00:21.581052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.581021    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"]
Nov 01 23:00:21.585938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.585909    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c54de0-29892900-5nnw7"]
Nov 01 23:00:21.619424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.619397    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"]
Nov 01 23:00:21.691765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.688334    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"]
Nov 01 23:00:21.693572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.693466    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-661e9f-29892900-kgfzd"]
Nov 01 23:00:21.749199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.749178    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:21.802019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.801988    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"]
Nov 01 23:00:21.809120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.809082    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-584561-29892900-hd6zx"]
Nov 01 23:00:21.837543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.837524    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:21.851799 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.851774    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca\") pod \"c727fe96-45a3-4928-b0b0-20a52737f04b\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") "
Nov 01 23:00:21.851906 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.851809    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj\" (UniqueName: \"kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj\") pod \"c727fe96-45a3-4928-b0b0-20a52737f04b\" (UID: \"c727fe96-45a3-4928-b0b0-20a52737f04b\") "
Nov 01 23:00:21.852092 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.852071    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca" pod "c727fe96-45a3-4928-b0b0-20a52737f04b" (UID: "c727fe96-45a3-4928-b0b0-20a52737f04b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:21.852479 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.852463    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c727fe96-45a3-4928-b0b0-20a52737f04b-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.853908 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.853890    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj" pod "c727fe96-45a3-4928-b0b0-20a52737f04b" (UID: "c727fe96-45a3-4928-b0b0-20a52737f04b"). InnerVolumeSpecName "kube-api-access-thnjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:21.905888 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.905862    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"]
Nov 01 23:00:21.913785 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.913761    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"]
Nov 01 23:00:21.934373 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.934333    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-71505e-29892900-2rlvm"]
Nov 01 23:00:21.953686 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.953666    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca\") pod \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") "
Nov 01 23:00:21.953819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.953697    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb\" (UniqueName: \"kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb\") pod \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\" (UID: \"bc33a895-3e28-4ab8-9e42-7076c0f9380d\") "
Nov 01 23:00:21.954117 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.954092    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca" pod "bc33a895-3e28-4ab8-9e42-7076c0f9380d" (UID: "bc33a895-3e28-4ab8-9e42-7076c0f9380d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:21.954242 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.954227    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thnjj\" (UniqueName: \"kubernetes.io/projected/c727fe96-45a3-4928-b0b0-20a52737f04b-kube-api-access-thnjj\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.954280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.954245    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc33a895-3e28-4ab8-9e42-7076c0f9380d-maas-api-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:21.956203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:21.956179    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb" pod "bc33a895-3e28-4ab8-9e42-7076c0f9380d" (UID: "bc33a895-3e28-4ab8-9e42-7076c0f9380d"). InnerVolumeSpecName "kube-api-access-v6lhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:22.006585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.006559    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"]
Nov 01 23:00:22.037512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.037483    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"]
Nov 01 23:00:22.055978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.055916    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6lhb\" (UniqueName: \"kubernetes.io/projected/bc33a895-3e28-4ab8-9e42-7076c0f9380d-kube-api-access-v6lhb\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:22.064495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.064472    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-8cc47a-29892900-wx8kf"]
Nov 01 23:00:22.226146 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.226111    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p" event={"ID":"c727fe96-45a3-4928-b0b0-20a52737f04b","Type":"ContainerDied","Data":"c641d345be3fa399205641b53421ae11c70d8072c9f6ae967539d912b4652d6d"}
Nov 01 23:00:22.226630 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.226172    4531 scope.go:122] "RemoveContainer" containerID="35f6012da2702e6e7d1c18a4b5f05d810b362bb8a3efa483f52677a5ec144ead"
Nov 01 23:00:22.226630 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.226283    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"
Nov 01 23:00:22.255226 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.255199    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"
Nov 01 23:00:22.255350 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.255228    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh" event={"ID":"bc33a895-3e28-4ab8-9e42-7076c0f9380d","Type":"ContainerDied","Data":"e0bacc1e55bd214336cf50cb84065d502ccc0abfa9bc89d005394766e86b6797"}
Nov 01 23:00:22.255350 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.255297    4531 scope.go:122] "RemoveContainer" containerID="c5326292f5a1e4a4b1887eee1f79317c9f7aa749415667d0b415e1f6a7a8a096"
Nov 01 23:00:22.358219 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.358144    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:22.650183 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.650100    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"]
Nov 01 23:00:22.658358 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.658322    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-7da9ed-29892900-5rt6p"]
Nov 01 23:00:22.681891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.681785    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:22.758561 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.758528    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"]
Nov 01 23:00:22.763039 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.763012    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-a6dd2f-29892900-x9xnh"]
Nov 01 23:00:22.820037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.820010    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" path="/var/lib/kubelet/pods/51b613b6-48e0-45ce-8cff-9daf0cecc43f/volumes"
Nov 01 23:00:22.820484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.820469    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" path="/var/lib/kubelet/pods/980b4aa0-cf34-4e16-b97f-417bc731da72/volumes"
Nov 01 23:00:22.820890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.820876    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" path="/var/lib/kubelet/pods/a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2/volumes"
Nov 01 23:00:22.821580 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.821565    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" path="/var/lib/kubelet/pods/a2cccee6-d93f-4447-af1f-db2de26c124b/volumes"
Nov 01 23:00:22.821970 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.821956    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" path="/var/lib/kubelet/pods/bc33a895-3e28-4ab8-9e42-7076c0f9380d/volumes"
Nov 01 23:00:22.822346 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.822333    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" path="/var/lib/kubelet/pods/c727fe96-45a3-4928-b0b0-20a52737f04b/volumes"
Nov 01 23:00:22.823002 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.822988    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" path="/var/lib/kubelet/pods/e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7/volumes"
Nov 01 23:00:22.976451 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:22.976417    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:23.248609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:23.248530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:23.449949 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:23.449918    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:23.680807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:23.680421    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:23.980774 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:23.978620    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:24.173707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.173675    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:24.351623 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.351532    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"]
Nov 01 23:00:24.354937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.354912    4531 cpu_manager.go:414] "containerMap: removing container" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.355118 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355103    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.355218 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355207    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:24.355302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355291    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:24.355396 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355386    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:24.355474 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355464    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:24.355552 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355542    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:24.355629 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355619    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:24.355715 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355705    4531 cpu_manager.go:414] "containerMap: removing container" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy"
Nov 01 23:00:24.355820 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355808    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy"
Nov 01 23:00:24.355974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.355962    4531 cpu_manager.go:414] "containerMap: removing container" podUID="86326a5d-e4f1-4038-9a92-453fa427443a" containerName="revoke-keys"
Nov 01 23:00:24.356065 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356052    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86326a5d-e4f1-4038-9a92-453fa427443a" containerName="revoke-keys"
Nov 01 23:00:24.356151 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356141    4531 cpu_manager.go:414] "containerMap: removing container" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.356236 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356225    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.356323 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356313    4531 cpu_manager.go:414] "containerMap: removing container" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:24.356402 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356392    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:24.356492 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356483    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2b8a2634-02cf-48b2-9679-037cb4a2783d" containerName="payload-processing"
Nov 01 23:00:24.356577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356566    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b8a2634-02cf-48b2-9679-037cb4a2783d" containerName="payload-processing"
Nov 01 23:00:24.356665 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356655    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:24.356771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356759    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:24.356867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356857    4531 cpu_manager.go:414] "containerMap: removing container" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.356943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.356933    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.357024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357015    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.357104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357094    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.357187 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357178    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.357253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357243    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.357334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357324    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.357409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357399    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.357490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357480    4531 cpu_manager.go:414] "containerMap: removing container" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.357570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357559    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.357661 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357651    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerName="cleanup"
Nov 01 23:00:24.357759 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357748    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerName="cleanup"
Nov 01 23:00:24.357850 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357840    4531 cpu_manager.go:414] "containerMap: removing container" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:24.357955 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.357944    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:24.358045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358035    4531 cpu_manager.go:414] "containerMap: removing container" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.358127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358117    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.358214 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358205    4531 cpu_manager.go:414] "containerMap: removing container" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.358290 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358280    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.358364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358355    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.358484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358471    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.358574 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358562    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:24.358655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358645    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:24.358762 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358751    4531 cpu_manager.go:414] "containerMap: removing container" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.358854 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358844    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.358940 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.358930    4531 cpu_manager.go:414] "containerMap: removing container" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.359147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.359133    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.359247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.359236    4531 cpu_manager.go:414] "containerMap: removing container" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.359652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.359636    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.359807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.359792    4531 cpu_manager.go:414] "containerMap: removing container" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.359903 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.359892    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.359991 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.359982    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.360078 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360068    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.360182 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360172    4531 cpu_manager.go:414] "containerMap: removing container" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.360262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360252    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.360350 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360340    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy"
Nov 01 23:00:24.360426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360416    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy"
Nov 01 23:00:24.360512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360502    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerName="cleanup"
Nov 01 23:00:24.360594 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360583    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerName="cleanup"
Nov 01 23:00:24.360670 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.360661    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:24.374501 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.374481    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:24.376358 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376338    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:24.376503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376488    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:24.376608 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376513    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:24.376608 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376521    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:24.376608 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376531    4531 cpu_manager.go:414] "containerMap: removing container" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.376846 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376613    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.376846 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376628    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1fa3111e-1565-4bcd-913a-c3161e31c140" containerName="payload-pre-processing"
Nov 01 23:00:24.376846 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.376638    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa3111e-1565-4bcd-913a-c3161e31c140" containerName="payload-pre-processing"
Nov 01 23:00:24.377045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.372657    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:24.377252 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.377096    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.377400 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.377386    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:24.377531 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.377517    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:24.377657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.377644    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:24.377784 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.377773    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="06062fa0-7733-4503-b2b6-10065547bf34" containerName="istio-proxy"
Nov 01 23:00:24.377910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.377899    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="86326a5d-e4f1-4038-9a92-453fa427443a" containerName="revoke-keys"
Nov 01 23:00:24.378016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378005    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.378138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378113    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:24.378247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378237    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b8a2634-02cf-48b2-9679-037cb4a2783d" containerName="payload-processing"
Nov 01 23:00:24.378364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378342    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:24.378468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378458    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.378565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378554    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.378708 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378695    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.378851 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378840    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.378987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.378954    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.379111 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379099    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc33a895-3e28-4ab8-9e42-7076c0f9380d" containerName="cleanup"
Nov 01 23:00:24.379255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379242    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerName="cleanup"
Nov 01 23:00:24.379444 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379424    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:24.379588 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379571    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:24.379753 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379717    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.379862 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379851    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:24.379957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.379948    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.380093 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380081    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.380197 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380187    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c727fe96-45a3-4928-b0b0-20a52737f04b" containerName="cleanup"
Nov 01 23:00:24.380302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380293    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:24.380404 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380394    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:24.380496 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380487    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3e3eed-c404-4205-b21f-fc5d8c982229" containerName="istio-proxy"
Nov 01 23:00:24.380620 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380611    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aec38feb-bb0d-49e6-bb55-5a5693ccca4d" containerName="cleanup"
Nov 01 23:00:24.380746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380715    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:24.380880 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.380865    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:24.381026 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.381015    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:24.381127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.381115    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="980b4aa0-cf34-4e16-b97f-417bc731da72" containerName="cleanup"
Nov 01 23:00:24.381256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.381244    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa3111e-1565-4bcd-913a-c3161e31c140" containerName="payload-pre-processing"
Nov 01 23:00:24.397766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.389931    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.408131 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.405517    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"]
Nov 01 23:00:24.451655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.451534    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:00:24.451655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.451627    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:00:24.535610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.535575    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.536024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.535637    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-265tl\" (UniqueName: \"kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.536024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.535842    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.637147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.637076    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.637147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.637122    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-265tl\" (UniqueName: \"kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.637318 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.637196    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.637933 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.637907    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.639478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.639450    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.691463 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.691430    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-265tl\" (UniqueName: \"kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:24.736077 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:24.736052    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:25.120562 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:25.120534    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"]
Nov 01 23:00:25.129967 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:25.124798    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bc9e97_890c_45f5_b66d_5122e9dd9fa7.slice/crio-7db3a1a6ce6d4c5ffdf187405381c5f1be0c334ba008ddbe6f53b05b84149794 WatchSource:0}: Error finding container 7db3a1a6ce6d4c5ffdf187405381c5f1be0c334ba008ddbe6f53b05b84149794: Status 404 returned error can't find the container with id 7db3a1a6ce6d4c5ffdf187405381c5f1be0c334ba008ddbe6f53b05b84149794
Nov 01 23:00:25.304540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:25.304509    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w" event={"ID":"90bc9e97-890c-45f5-b66d-5122e9dd9fa7","Type":"ContainerStarted","Data":"7db3a1a6ce6d4c5ffdf187405381c5f1be0c334ba008ddbe6f53b05b84149794"}
Nov 01 23:00:25.804426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:25.804388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:26.317350 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:26.317315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w" event={"ID":"90bc9e97-890c-45f5-b66d-5122e9dd9fa7","Type":"ContainerStarted","Data":"a8d2cb7cc03f1ce725d5b127d1fc5285e7595e552d81a9f809aca7405d3f016c"}
Nov 01 23:00:26.363807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:26.363778    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:26.427939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:26.427892    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w" podStartSLOduration=2.42787861 podStartE2EDuration="2.42787861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:26.425411769 +0000 UTC m=+2207.743061771" watchObservedRunningTime="2026-11-01 23:00:26.42787861 +0000 UTC m=+2207.745528649"
Nov 01 23:00:26.956386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:26.956342    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr" podUID="67b8ec40-75be-4e2d-b263-f69b487c41ec" containerName="payload-processing" containerID="cri-o://d7abf28934cd61f8b5a66458ca562e8466fe4850c81c708202352f2b7bd89641" gracePeriod=25
Nov 01 23:00:26.957710 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:26.957505    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:27.022060 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.022022    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l" podUID="fbf3a948-317a-47d9-aef5-81303c4f5400" containerName="payload-pre-processing" containerID="cri-o://303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89" gracePeriod=25
Nov 01 23:00:27.166562 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:27.166516    4531 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf3a948_317a_47d9_aef5_81303c4f5400.slice/crio-303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf3a948_317a_47d9_aef5_81303c4f5400.slice/crio-conmon-303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 23:00:27.222143 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.222055    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:27.336259 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.336088    4531 generic.go:356] "Generic (PLEG): container finished" podID="67b8ec40-75be-4e2d-b263-f69b487c41ec" containerID="d7abf28934cd61f8b5a66458ca562e8466fe4850c81c708202352f2b7bd89641" exitCode=0
Nov 01 23:00:27.336259 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.336176    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr" event={"ID":"67b8ec40-75be-4e2d-b263-f69b487c41ec","Type":"ContainerDied","Data":"d7abf28934cd61f8b5a66458ca562e8466fe4850c81c708202352f2b7bd89641"}
Nov 01 23:00:27.338239 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.338214    4531 generic.go:356] "Generic (PLEG): container finished" podID="fbf3a948-317a-47d9-aef5-81303c4f5400" containerID="303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89" exitCode=0
Nov 01 23:00:27.338361 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.338332    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l" event={"ID":"fbf3a948-317a-47d9-aef5-81303c4f5400","Type":"ContainerDied","Data":"303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89"}
Nov 01 23:00:27.340107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.339980    4531 generic.go:356] "Generic (PLEG): container finished" podID="90bc9e97-890c-45f5-b66d-5122e9dd9fa7" containerID="a8d2cb7cc03f1ce725d5b127d1fc5285e7595e552d81a9f809aca7405d3f016c" exitCode=0
Nov 01 23:00:27.340107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.340059    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w" event={"ID":"90bc9e97-890c-45f5-b66d-5122e9dd9fa7","Type":"ContainerDied","Data":"a8d2cb7cc03f1ce725d5b127d1fc5285e7595e552d81a9f809aca7405d3f016c"}
Nov 01 23:00:27.404449 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.403718    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 23:00:27.445133 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.445111    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 23:00:27.506295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.506210    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume\") pod \"fbf3a948-317a-47d9-aef5-81303c4f5400\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") "
Nov 01 23:00:27.506295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.506262    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg\" (UniqueName: \"kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg\") pod \"fbf3a948-317a-47d9-aef5-81303c4f5400\" (UID: \"fbf3a948-317a-47d9-aef5-81303c4f5400\") "
Nov 01 23:00:27.506521 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.506366    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume\") pod \"67b8ec40-75be-4e2d-b263-f69b487c41ec\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") "
Nov 01 23:00:27.506521 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.506436    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q\" (UniqueName: \"kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q\") pod \"67b8ec40-75be-4e2d-b263-f69b487c41ec\" (UID: \"67b8ec40-75be-4e2d-b263-f69b487c41ec\") "
Nov 01 23:00:27.507953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.507920    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume" pod "67b8ec40-75be-4e2d-b263-f69b487c41ec" (UID: "67b8ec40-75be-4e2d-b263-f69b487c41ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:27.508126 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.508106    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume" pod "fbf3a948-317a-47d9-aef5-81303c4f5400" (UID: "fbf3a948-317a-47d9-aef5-81303c4f5400"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:27.509690 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.509666    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q" pod "67b8ec40-75be-4e2d-b263-f69b487c41ec" (UID: "67b8ec40-75be-4e2d-b263-f69b487c41ec"). InnerVolumeSpecName "kube-api-access-vww6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:27.510543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.510444    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg" pod "fbf3a948-317a-47d9-aef5-81303c4f5400" (UID: "fbf3a948-317a-47d9-aef5-81303c4f5400"). InnerVolumeSpecName "kube-api-access-tgqsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:27.525650 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.525619    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:27.608663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.608191    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbf3a948-317a-47d9-aef5-81303c4f5400-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:27.608663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.608217    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgqsg\" (UniqueName: \"kubernetes.io/projected/fbf3a948-317a-47d9-aef5-81303c4f5400-kube-api-access-tgqsg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:27.608663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.608227    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b8ec40-75be-4e2d-b263-f69b487c41ec-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:27.608663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.608235    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vww6q\" (UniqueName: \"kubernetes.io/projected/67b8ec40-75be-4e2d-b263-f69b487c41ec-kube-api-access-vww6q\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:27.795985 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.795895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:27.999285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:27.999241    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:28.353340 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.353306    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr" event={"ID":"67b8ec40-75be-4e2d-b263-f69b487c41ec","Type":"ContainerDied","Data":"11f215fa5bf1645a4ffcb5487dfd3264598f769290e4d502fb59aae3808088f3"}
Nov 01 23:00:28.353532 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.353383    4531 scope.go:122] "RemoveContainer" containerID="d7abf28934cd61f8b5a66458ca562e8466fe4850c81c708202352f2b7bd89641"
Nov 01 23:00:28.353532 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.353311    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"
Nov 01 23:00:28.355049 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.355034    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"
Nov 01 23:00:28.355152 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.355082    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l" event={"ID":"fbf3a948-317a-47d9-aef5-81303c4f5400","Type":"ContainerDied","Data":"9945753408fa04d0d798dc03533276e5c7456eb593af5b80d2afac47231ceb61"}
Nov 01 23:00:28.370165 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.370143    4531 scope.go:122] "RemoveContainer" containerID="303011ff6f365e9e236d28fdde0fd2d6030064e400b889750aac806cf1b69b89"
Nov 01 23:00:28.415597 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.415570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:28.521265 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.521236    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"]
Nov 01 23:00:28.528265 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.527337    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-71505e-985f9db67-t448l"]
Nov 01 23:00:28.576788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.576761    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:28.624655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.624579    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca\") pod \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") "
Nov 01 23:00:28.624655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.624630    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl\" (UniqueName: \"kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl\") pod \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") "
Nov 01 23:00:28.624899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.624826    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca\") pod \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\" (UID: \"90bc9e97-890c-45f5-b66d-5122e9dd9fa7\") "
Nov 01 23:00:28.625162 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.625140    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca" pod "90bc9e97-890c-45f5-b66d-5122e9dd9fa7" (UID: "90bc9e97-890c-45f5-b66d-5122e9dd9fa7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:28.625458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.625437    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.627172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.627133    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl" pod "90bc9e97-890c-45f5-b66d-5122e9dd9fa7" (UID: "90bc9e97-890c-45f5-b66d-5122e9dd9fa7"). InnerVolumeSpecName "kube-api-access-265tl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:28.627408 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.627382    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca" pod "90bc9e97-890c-45f5-b66d-5122e9dd9fa7" (UID: "90bc9e97-890c-45f5-b66d-5122e9dd9fa7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:28.628175 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.628119    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"]
Nov 01 23:00:28.651549 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.651520    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-71505e-5c55d7dd4f-ffwjr"]
Nov 01 23:00:28.727072 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.727040    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.727225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.727089    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-265tl\" (UniqueName: \"kubernetes.io/projected/90bc9e97-890c-45f5-b66d-5122e9dd9fa7-kube-api-access-265tl\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:28.820020 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.819982    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b8ec40-75be-4e2d-b263-f69b487c41ec" path="/var/lib/kubelet/pods/67b8ec40-75be-4e2d-b263-f69b487c41ec/volumes"
Nov 01 23:00:28.820561 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:28.820539    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf3a948-317a-47d9-aef5-81303c4f5400" path="/var/lib/kubelet/pods/fbf3a948-317a-47d9-aef5-81303c4f5400/volumes"
Nov 01 23:00:29.367090 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:29.367062    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"
Nov 01 23:00:29.368894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:29.367326    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w" event={"ID":"90bc9e97-890c-45f5-b66d-5122e9dd9fa7","Type":"ContainerDied","Data":"7db3a1a6ce6d4c5ffdf187405381c5f1be0c334ba008ddbe6f53b05b84149794"}
Nov 01 23:00:29.368894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:29.368701    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7db3a1a6ce6d4c5ffdf187405381c5f1be0c334ba008ddbe6f53b05b84149794"
Nov 01 23:00:31.056245 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:31.056206    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:31.311666 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:31.311577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:31.522751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:31.522375    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:32.860792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:32.853937    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"]
Nov 01 23:00:33.310554 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.310508    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"]
Nov 01 23:00:33.408197 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.408147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:33.542907 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.531302    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": context deadline exceeded"
Nov 01 23:00:33.601147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.601032    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.91:8000/ready\": context deadline exceeded"
Nov 01 23:00:33.667776 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.661290    4531 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:33.674200 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.674155    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:33.892804 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.887689    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:33.905173 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.905128    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:33.958259 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:33.944811    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:35.777679 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:35.777590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:35.988747 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:35.988514    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:36.032411 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:36.032324    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"]
Nov 01 23:00:36.032872 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:36.032838    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" containerID="cri-o://ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0" gracePeriod=30
Nov 01 23:00:36.242468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:36.242427    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:36.442700 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:36.442612    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:37.854643 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:37.854587    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerName="payload-processing" containerID="cri-o://21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac" gracePeriod=25
Nov 01 23:00:38.124356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:38.124273    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:38.311807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:38.311765    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerName="payload-pre-processing" containerID="cri-o://b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8" gracePeriod=25
Nov 01 23:00:39.192884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:39.192849    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:39.512051 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:39.512009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:39.608225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:39.608185    4531 patch_prober.go:29] interesting pod/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:00:39.608424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:39.608254    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:00:39.713958 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:39.713916    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:39.916769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:39.916653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:40.111035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.110982    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:00:40.111241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.111077    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:00:40.120752 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.120570    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:40.338299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.338253    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:40.930112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.930076    4531 scope.go:122] "RemoveContainer" containerID="bd76a7ba249f168542ddbf9df0c427204daef4a2057f513ab8627cd055703f1c"
Nov 01 23:00:40.947303 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.947281    4531 scope.go:122] "RemoveContainer" containerID="8b05f3d53f5e5b4aeb44e20a6a11e7e8f5e3e82085faa8e72df4f71791ccb527"
Nov 01 23:00:40.959646 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:40.959614    4531 scope.go:122] "RemoveContainer" containerID="81d9d52075b4fcbfc69a3476b9ab728240b83fbaf0f1b3c3406e8b654b8bf8be"
Nov 01 23:00:41.536691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.536298    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695168    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695267    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695369    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695408    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695466    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695558    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695604    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695668    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.695749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.695699    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs\") pod \"6c400087-2db5-4bbd-9427-be06e9acbf8f\" (UID: \"6c400087-2db5-4bbd-9427-be06e9acbf8f\") "
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.696366    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.696919    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.697136    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.697329    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.697924    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.697948    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.697962    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c400087-2db5-4bbd-9427-be06e9acbf8f-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.697975    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.698049    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.701004    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.705878    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.705920    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "kube-api-access-cfcmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:41.711750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.705947    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy" pod "6c400087-2db5-4bbd-9427-be06e9acbf8f" (UID: "6c400087-2db5-4bbd-9427-be06e9acbf8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:00:41.799153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.799061    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.799153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.799095    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c400087-2db5-4bbd-9427-be06e9acbf8f-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.799153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.799107    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfcmq\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-kube-api-access-cfcmq\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.799153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.799116    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:41.799153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:41.799125    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c400087-2db5-4bbd-9427-be06e9acbf8f-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:42.064771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.064670    4531 generic.go:356] "Generic (PLEG): container finished" podID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerID="ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0" exitCode=0
Nov 01 23:00:42.065057 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.064758    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" event={"ID":"6c400087-2db5-4bbd-9427-be06e9acbf8f","Type":"ContainerDied","Data":"ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0"}
Nov 01 23:00:42.065057 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.064810    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"
Nov 01 23:00:42.065057 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.064839    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv" event={"ID":"6c400087-2db5-4bbd-9427-be06e9acbf8f","Type":"ContainerDied","Data":"13296f788da8978e2c6a7b06e44c9f8409db3614b10c63772a656289784b1589"}
Nov 01 23:00:42.065057 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.064890    4531 scope.go:122] "RemoveContainer" containerID="ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0"
Nov 01 23:00:42.082900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.082876    4531 scope.go:122] "RemoveContainer" containerID="11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"
Nov 01 23:00:42.098484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.098463    4531 scope.go:122] "RemoveContainer" containerID="ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0"
Nov 01 23:00:42.099707 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:42.099676    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0\": container with ID starting with ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0 not found: ID does not exist" containerID="ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0"
Nov 01 23:00:42.100167 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.099717    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0"} err="failed to get container status \"ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0\": rpc error: code = NotFound desc = could not find container \"ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0\": container with ID starting with ffe6ca16e4c1c6f6468ee8d7b775d445f92fdc91a6876c00fd70f8c962fe6ae0 not found: ID does not exist"
Nov 01 23:00:42.100266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.100169    4531 scope.go:122] "RemoveContainer" containerID="11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"
Nov 01 23:00:42.100718 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:42.100691    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b\": container with ID starting with 11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b not found: ID does not exist" containerID="11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"
Nov 01 23:00:42.100718 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.100774    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b"} err="failed to get container status \"11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b\": rpc error: code = NotFound desc = could not find container \"11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b\": container with ID starting with 11ce5fd3477770c0e8aad06b3d3d8e74416f976cb6c1358df47f0b6f8f27124b not found: ID does not exist"
Nov 01 23:00:42.225496 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.225459    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"]
Nov 01 23:00:42.267549 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.267508    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-71505e-openshift-default-75d49cbc4c-ckgpv"]
Nov 01 23:00:42.534157 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:42.534024    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:00:42.819528 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:42.819439    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" path="/var/lib/kubelet/pods/6c400087-2db5-4bbd-9427-be06e9acbf8f/volumes"
Nov 01 23:00:43.081254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.081196    4531 generic.go:356] "Generic (PLEG): container finished" podID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerID="4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb" exitCode=0
Nov 01 23:00:43.081254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.081249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" event={"ID":"290646e6-1e8b-41ff-9e0b-8d21674fbf9c","Type":"ContainerDied","Data":"4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"}
Nov 01 23:00:43.081627 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.081611    4531 scope.go:122] "RemoveContainer" containerID="4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"
Nov 01 23:00:43.083255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.083225    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:00:43.083351 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.083298    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:00:43.083351 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.083333    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:00:43.823887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.823848    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:43.974602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.974574    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"]
Nov 01 23:00:43.975388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975370    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fbf3a948-317a-47d9-aef5-81303c4f5400" containerName="payload-pre-processing"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975390    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf3a948-317a-47d9-aef5-81303c4f5400" containerName="payload-pre-processing"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975420    4531 cpu_manager.go:414] "containerMap: removing container" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975426    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975436    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975443    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975460    4531 cpu_manager.go:414] "containerMap: removing container" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy"
Nov 01 23:00:43.975470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975465    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975494    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975502    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975518    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975526    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975537    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975544    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975557    4531 cpu_manager.go:414] "containerMap: removing container" podUID="67b8ec40-75be-4e2d-b263-f69b487c41ec" containerName="payload-processing"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975562    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b8ec40-75be-4e2d-b263-f69b487c41ec" containerName="payload-processing"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975569    4531 cpu_manager.go:414] "containerMap: removing container" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975574    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39b0859f-d187-4896-9e50-f9d981010d7d" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975587    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975591    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975599    4531 cpu_manager.go:414] "containerMap: removing container" podUID="90bc9e97-890c-45f5-b66d-5122e9dd9fa7" containerName="revoke-keys"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975605    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bc9e97-890c-45f5-b66d-5122e9dd9fa7" containerName="revoke-keys"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975772    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bc9e97-890c-45f5-b66d-5122e9dd9fa7" containerName="revoke-keys"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975791    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf3a948-317a-47d9-aef5-81303c4f5400" containerName="payload-pre-processing"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975808    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975819    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c400087-2db5-4bbd-9427-be06e9acbf8f" containerName="istio-proxy"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975828    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b613b6-48e0-45ce-8cff-9daf0cecc43f" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975836    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="99225305-c4f2-4987-9fcd-833f501913a8" containerName="cleanup"
Nov 01 23:00:43.975849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975849    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ac75d0-c2d9-4ee4-9ebb-0014c82368b2" containerName="cleanup"
Nov 01 23:00:43.976585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975878    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="086dc50c-8e76-437f-af84-ca77b63d7228" containerName="cleanup"
Nov 01 23:00:43.976585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975889    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec531e1a-fb93-4b56-afed-7a77461092cd" containerName="cleanup"
Nov 01 23:00:43.976585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975899    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6ffee94-ed10-4a3a-b3fb-856cfc4b36a7" containerName="cleanup"
Nov 01 23:00:43.976585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975910    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2cccee6-d93f-4447-af1f-db2de26c124b" containerName="cleanup"
Nov 01 23:00:43.976585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.975919    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b8ec40-75be-4e2d-b263-f69b487c41ec" containerName="payload-processing"
Nov 01 23:00:43.979809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:43.979787    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.029225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.029189    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw7cr\" (UniqueName: \"kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.029349 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.029315    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.029415 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.029397    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.029647 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.029627    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:00:44.029698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.029682    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:00:44.033234 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.033208    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:44.039476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.039453    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"]
Nov 01 23:00:44.078565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.071050    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"]
Nov 01 23:00:44.078738 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.078697    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.109961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.109928    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" event={"ID":"290646e6-1e8b-41ff-9e0b-8d21674fbf9c","Type":"ContainerStarted","Data":"ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7"}
Nov 01 23:00:44.130658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.130626    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw7cr\" (UniqueName: \"kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.130802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.130782    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.130878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.130820    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.131177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.130941    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.131177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.131001    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.131177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.131094    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78dh9\" (UniqueName: \"kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.131632 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.131613    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.133348 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.133320    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.146120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.146096    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"]
Nov 01 23:00:44.185641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.185611    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw7cr\" (UniqueName: \"kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr\") pod \"maas-api-revoke-sub-359228b8-p7gwp\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.231909 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.231871    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.232118 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.231937    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78dh9\" (UniqueName: \"kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.232189 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.232147    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.232636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.232602    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.240246 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.240117    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.248751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.245330    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:44.296977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.296939    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78dh9\" (UniqueName: \"kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9\") pod \"maas-api-revoke-sub-5d21a208-n9pxf\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.298920 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.298895    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:44.411561 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.410911    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:44.709053 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.709017    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"]
Nov 01 23:00:44.714012 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:44.713981    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a4f430_c80c_4dc6_aadf_f75e7ee47d5b.slice/crio-bc0514308d3842bfb9075eca6da2e71d8e5966a3d3f773642b6146f7ca0f52c5 WatchSource:0}: Error finding container bc0514308d3842bfb9075eca6da2e71d8e5966a3d3f773642b6146f7ca0f52c5: Status 404 returned error can't find the container with id bc0514308d3842bfb9075eca6da2e71d8e5966a3d3f773642b6146f7ca0f52c5
Nov 01 23:00:44.895892 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.895694    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"]
Nov 01 23:00:44.972831 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:44.967671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:45.068304 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:45.068274    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 23:00:45.068748 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:45.068709    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 01 23:00:45.068924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:45.068900    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 01 23:00:45.119425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:45.119395    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp" event={"ID":"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b","Type":"ContainerStarted","Data":"bc0514308d3842bfb9075eca6da2e71d8e5966a3d3f773642b6146f7ca0f52c5"}
Nov 01 23:00:45.121400 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:45.121375    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf" event={"ID":"cc73e051-4f5e-4416-bc8b-65183c0eb84e","Type":"ContainerStarted","Data":"3502c521fbefdc581b07ba296d05c945452bd7c2c247124a19f3f61cd044d7cf"}
Nov 01 23:00:46.082522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:46.082059    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 01 23:00:46.082522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:46.082114    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 01 23:00:46.130108 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:46.130081    4531 generic.go:356] "Generic (PLEG): container finished" podID="32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" containerID="e9c428d95eb05d8abbd598b43ea6a20654a0d5cdb05c7c282ee5a4e45d8209f8" exitCode=0
Nov 01 23:00:46.130262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:46.130161    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp" event={"ID":"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b","Type":"ContainerDied","Data":"e9c428d95eb05d8abbd598b43ea6a20654a0d5cdb05c7c282ee5a4e45d8209f8"}
Nov 01 23:00:46.131671 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:46.131651    4531 generic.go:356] "Generic (PLEG): container finished" podID="cc73e051-4f5e-4416-bc8b-65183c0eb84e" containerID="a23ee382d2da97689dddb5cba239e2a3f75c6c320fc86c5985454e0895b6d721" exitCode=0
Nov 01 23:00:46.131793 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:46.131675    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf" event={"ID":"cc73e051-4f5e-4416-bc8b-65183c0eb84e","Type":"ContainerDied","Data":"a23ee382d2da97689dddb5cba239e2a3f75c6c320fc86c5985454e0895b6d721"}
Nov 01 23:00:47.069757 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.065046    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 01 23:00:47.069757 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.065106    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 01 23:00:47.238147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.238089    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:00:47.239667 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.238482    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:00:47.457885 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.457858    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:47.489955 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.489927    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:47.614139 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614068    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca\") pod \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") "
Nov 01 23:00:47.614302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614281    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca\") pod \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") "
Nov 01 23:00:47.614406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614388    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca\") pod \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") "
Nov 01 23:00:47.614463 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614423    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca\") pod \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") "
Nov 01 23:00:47.614510 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614494    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr\" (UniqueName: \"kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr\") pod \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\" (UID: \"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b\") "
Nov 01 23:00:47.614567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614549    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9\" (UniqueName: \"kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9\") pod \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\" (UID: \"cc73e051-4f5e-4416-bc8b-65183c0eb84e\") "
Nov 01 23:00:47.614880 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca" pod "cc73e051-4f5e-4416-bc8b-65183c0eb84e" (UID: "cc73e051-4f5e-4416-bc8b-65183c0eb84e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:47.615130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.614972    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca" pod "32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" (UID: "32a4f430-c80c-4dc6-aadf-f75e7ee47d5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:47.615321 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.615293    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:47.615401 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.615327    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:47.616885 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.616841    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca" pod "32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" (UID: "32a4f430-c80c-4dc6-aadf-f75e7ee47d5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:47.618386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.618151    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca" pod "cc73e051-4f5e-4416-bc8b-65183c0eb84e" (UID: "cc73e051-4f5e-4416-bc8b-65183c0eb84e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:47.618386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.618343    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9" pod "cc73e051-4f5e-4416-bc8b-65183c0eb84e" (UID: "cc73e051-4f5e-4416-bc8b-65183c0eb84e"). InnerVolumeSpecName "kube-api-access-78dh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:47.618592 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.618577    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr" pod "32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" (UID: "32a4f430-c80c-4dc6-aadf-f75e7ee47d5b"). InnerVolumeSpecName "kube-api-access-qw7cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:47.716828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.716799    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78dh9\" (UniqueName: \"kubernetes.io/projected/cc73e051-4f5e-4416-bc8b-65183c0eb84e-kube-api-access-78dh9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:47.717010 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.716998    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:47.717084 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.717076    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc73e051-4f5e-4416-bc8b-65183c0eb84e-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:47.717136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:47.717129    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qw7cr\" (UniqueName: \"kubernetes.io/projected/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b-kube-api-access-qw7cr\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:48.065115 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.065082    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 01 23:00:48.065386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.065358    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 01 23:00:48.150423 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.150399    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"
Nov 01 23:00:48.150704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.150417    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp" event={"ID":"32a4f430-c80c-4dc6-aadf-f75e7ee47d5b","Type":"ContainerDied","Data":"bc0514308d3842bfb9075eca6da2e71d8e5966a3d3f773642b6146f7ca0f52c5"}
Nov 01 23:00:48.150801 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.150759    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc0514308d3842bfb9075eca6da2e71d8e5966a3d3f773642b6146f7ca0f52c5"
Nov 01 23:00:48.151891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.151868    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf" event={"ID":"cc73e051-4f5e-4416-bc8b-65183c0eb84e","Type":"ContainerDied","Data":"3502c521fbefdc581b07ba296d05c945452bd7c2c247124a19f3f61cd044d7cf"}
Nov 01 23:00:48.151987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.151899    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3502c521fbefdc581b07ba296d05c945452bd7c2c247124a19f3f61cd044d7cf"
Nov 01 23:00:48.151987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:48.151937    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"
Nov 01 23:00:49.079751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.077028    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused" start-of-body=
Nov 01 23:00:49.079751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.077083    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.180:15021/healthz/ready\": dial tcp 10.133.0.180:15021: connect: connection refused"
Nov 01 23:00:49.594467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.594397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:49.835437 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.835405    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:49.961566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.961519    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"]
Nov 01 23:00:49.962687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.962659    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cc73e051-4f5e-4416-bc8b-65183c0eb84e" containerName="revoke-keys"
Nov 01 23:00:49.962687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.962687    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc73e051-4f5e-4416-bc8b-65183c0eb84e" containerName="revoke-keys"
Nov 01 23:00:49.962907 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.962743    4531 cpu_manager.go:414] "containerMap: removing container" podUID="32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" containerName="revoke-keys"
Nov 01 23:00:49.962907 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.962752    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" containerName="revoke-keys"
Nov 01 23:00:49.963082 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.962939    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" containerName="revoke-keys"
Nov 01 23:00:49.963082 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.962970    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc73e051-4f5e-4416-bc8b-65183c0eb84e" containerName="revoke-keys"
Nov 01 23:00:49.967867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:49.967840    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.018757 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.017782    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"]
Nov 01 23:00:50.044958 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.044922    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdvf\" (UniqueName: \"kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf\") pod \"payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.045114 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.045044    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.058667 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.058648    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aa47d8-dockercfg-h7nfl\""
Nov 01 23:00:50.059901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.059879    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aa47d8\""
Nov 01 23:00:50.068173 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.068147    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 23:00:50.068594 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.068566    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 23:00:50.069531 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.069512    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 23:00:50.146928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.146900    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdvf\" (UniqueName: \"kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf\") pod \"payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.147422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.147010    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.148643 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.148361    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.215417 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.215131    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdvf\" (UniqueName: \"kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf\") pod \"payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.285688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.285655    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:50.522028 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.521930    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:50.672690 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:50.672651    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"]
Nov 01 23:00:50.677095 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:50.677063    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2af3b9b_1f8a_4ecd_b3bf_16f5ee1f8d6f.slice/crio-de1745ddee242853c73ea994c654942d3cbd541620c7d19090756850c8196ff3 WatchSource:0}: Error finding container de1745ddee242853c73ea994c654942d3cbd541620c7d19090756850c8196ff3: Status 404 returned error can't find the container with id de1745ddee242853c73ea994c654942d3cbd541620c7d19090756850c8196ff3
Nov 01 23:00:51.181289 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:51.181249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" event={"ID":"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f","Type":"ContainerStarted","Data":"8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c"}
Nov 01 23:00:51.181687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:51.181315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" event={"ID":"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f","Type":"ContainerStarted","Data":"de1745ddee242853c73ea994c654942d3cbd541620c7d19090756850c8196ff3"}
Nov 01 23:00:51.181687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:51.181344    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:00:51.290101 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:51.290035    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" podStartSLOduration=2.290017432 podStartE2EDuration="2.290017432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:00:51.287275466 +0000 UTC m=+2232.604925467" watchObservedRunningTime="2026-11-01 23:00:51.290017432 +0000 UTC m=+2232.607667435"
Nov 01 23:00:55.530791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:55.530753    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:56.290051 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.290021    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"]
Nov 01 23:00:56.294847 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.294822    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.348757 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.348710    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"]
Nov 01 23:00:56.388861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.388828    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:00:56.388861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.388836    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:00:56.445954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.445928    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.446088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.446041    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.446132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.446096    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t5qp\" (UniqueName: \"kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.547506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.547444    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t5qp\" (UniqueName: \"kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.547901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.547579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.547901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.547642    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.548386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.548364    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.550016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.549990    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.603766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.603739    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t5qp\" (UniqueName: \"kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp\") pod \"maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.610405 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.610385    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:56.984149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:56.984120    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"]
Nov 01 23:00:56.987432 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:00:56.987402    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e89e194_2302_4692_94ca_ee9b46406c04.slice/crio-87669f23021d7b6abbdec31dbfe00309f07cf9ba97f50c150ac5af143ae1ddbd WatchSource:0}: Error finding container 87669f23021d7b6abbdec31dbfe00309f07cf9ba97f50c150ac5af143ae1ddbd: Status 404 returned error can't find the container with id 87669f23021d7b6abbdec31dbfe00309f07cf9ba97f50c150ac5af143ae1ddbd
Nov 01 23:00:57.237965 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:57.237882    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:00:57.238203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:57.238178    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:00:57.240428 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:57.240403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz" event={"ID":"8e89e194-2302-4692-94ca-ee9b46406c04","Type":"ContainerStarted","Data":"87669f23021d7b6abbdec31dbfe00309f07cf9ba97f50c150ac5af143ae1ddbd"}
Nov 01 23:00:58.122310 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.122289    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 23:00:58.254851 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.254818    4531 generic.go:356] "Generic (PLEG): container finished" podID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerID="21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac" exitCode=0
Nov 01 23:00:58.255050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.254881    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"
Nov 01 23:00:58.255050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.254940    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" event={"ID":"f33ff84c-4c87-4ec2-9a33-25cef07051a0","Type":"ContainerDied","Data":"21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac"}
Nov 01 23:00:58.255050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.255027    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s" event={"ID":"f33ff84c-4c87-4ec2-9a33-25cef07051a0","Type":"ContainerDied","Data":"a4a9125328798e5a0478368b41e914a5543b053216171460eebecbce30de74a9"}
Nov 01 23:00:58.255255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.255079    4531 scope.go:122] "RemoveContainer" containerID="21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac"
Nov 01 23:00:58.256860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.256804    4531 generic.go:356] "Generic (PLEG): container finished" podID="8e89e194-2302-4692-94ca-ee9b46406c04" containerID="7f24663084b5f4be0d73f2215f60a322ae7c296f04d9b8d5baa23c837c8fd196" exitCode=0
Nov 01 23:00:58.256956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.256913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz" event={"ID":"8e89e194-2302-4692-94ca-ee9b46406c04","Type":"ContainerDied","Data":"7f24663084b5f4be0d73f2215f60a322ae7c296f04d9b8d5baa23c837c8fd196"}
Nov 01 23:00:58.268703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.268675    4531 scope.go:122] "RemoveContainer" containerID="21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac"
Nov 01 23:00:58.269032 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:58.269011    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac\": container with ID starting with 21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac not found: ID does not exist" containerID="21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac"
Nov 01 23:00:58.269119 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.269041    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac"} err="failed to get container status \"21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac\": rpc error: code = NotFound desc = could not find container \"21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac\": container with ID starting with 21e4c0be112b5949b977549696d389972b4962f0f1a59dcaae0d0c4556cdb0ac not found: ID does not exist"
Nov 01 23:00:58.269303 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.269278    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume\") pod \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") "
Nov 01 23:00:58.269392 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.269375    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv\" (UniqueName: \"kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv\") pod \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\" (UID: \"f33ff84c-4c87-4ec2-9a33-25cef07051a0\") "
Nov 01 23:00:58.270919 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.270891    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume" pod "f33ff84c-4c87-4ec2-9a33-25cef07051a0" (UID: "f33ff84c-4c87-4ec2-9a33-25cef07051a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:58.272887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.272866    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv" pod "f33ff84c-4c87-4ec2-9a33-25cef07051a0" (UID: "f33ff84c-4c87-4ec2-9a33-25cef07051a0"). InnerVolumeSpecName "kube-api-access-zwdpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:58.371663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.371608    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f33ff84c-4c87-4ec2-9a33-25cef07051a0-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:58.371663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.371631    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwdpv\" (UniqueName: \"kubernetes.io/projected/f33ff84c-4c87-4ec2-9a33-25cef07051a0-kube-api-access-zwdpv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:58.587619 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.587595    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 23:00:58.674928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.674874    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk\" (UniqueName: \"kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk\") pod \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") "
Nov 01 23:00:58.674928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.674906    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume\") pod \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\" (UID: \"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21\") "
Nov 01 23:00:58.675288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.675266    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume" pod "1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" (UID: "1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:58.676926 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.676904    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk" pod "1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" (UID: "1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21"). InnerVolumeSpecName "kube-api-access-qqkrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:58.689794 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.689764    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"]
Nov 01 23:00:58.693913 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.693891    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-5b8fd2-544789f87-55t7s"]
Nov 01 23:00:58.776139 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.776115    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqkrk\" (UniqueName: \"kubernetes.io/projected/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-kube-api-access-qqkrk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:58.776139 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.776137    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:58.822306 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:58.822275    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" path="/var/lib/kubelet/pods/f33ff84c-4c87-4ec2-9a33-25cef07051a0/volumes"
Nov 01 23:00:59.209189 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.209125    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:59.272389 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.272354    4531 generic.go:356] "Generic (PLEG): container finished" podID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerID="b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8" exitCode=0
Nov 01 23:00:59.272575 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.272402    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"
Nov 01 23:00:59.272575 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.272430    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" event={"ID":"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21","Type":"ContainerDied","Data":"b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8"}
Nov 01 23:00:59.272711 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.272582    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf" event={"ID":"1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21","Type":"ContainerDied","Data":"436fd06577cb4cdce8af429fc76a835a59c8ee1f230ec1e74376f38908edeff6"}
Nov 01 23:00:59.272711 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.272613    4531 scope.go:122] "RemoveContainer" containerID="b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8"
Nov 01 23:00:59.292457 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.292423    4531 scope.go:122] "RemoveContainer" containerID="b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8"
Nov 01 23:00:59.292869 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:00:59.292844    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8\": container with ID starting with b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8 not found: ID does not exist" containerID="b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8"
Nov 01 23:00:59.292964 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.292879    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8"} err="failed to get container status \"b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8\": rpc error: code = NotFound desc = could not find container \"b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8\": container with ID starting with b73241a189d8e408ba9c2dfca8d55c1c5e9f9e21504717c9c95e9954c9536ea8 not found: ID does not exist"
Nov 01 23:00:59.401066 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.401030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:00:59.432094 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.432066    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"]
Nov 01 23:00:59.436312 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.436290    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-5b8fd2-7d988466c9-4s5sf"]
Nov 01 23:00:59.456987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.456964    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:00:59.586554 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.586466    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca\") pod \"8e89e194-2302-4692-94ca-ee9b46406c04\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") "
Nov 01 23:00:59.586554 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.586533    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca\") pod \"8e89e194-2302-4692-94ca-ee9b46406c04\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") "
Nov 01 23:00:59.586826 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.586629    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp\" (UniqueName: \"kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp\") pod \"8e89e194-2302-4692-94ca-ee9b46406c04\" (UID: \"8e89e194-2302-4692-94ca-ee9b46406c04\") "
Nov 01 23:00:59.586959 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.586926    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca" pod "8e89e194-2302-4692-94ca-ee9b46406c04" (UID: "8e89e194-2302-4692-94ca-ee9b46406c04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:00:59.587277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.587247    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:59.589125 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.589096    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp" pod "8e89e194-2302-4692-94ca-ee9b46406c04" (UID: "8e89e194-2302-4692-94ca-ee9b46406c04"). InnerVolumeSpecName "kube-api-access-8t5qp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:00:59.589768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.589737    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca" pod "8e89e194-2302-4692-94ca-ee9b46406c04" (UID: "8e89e194-2302-4692-94ca-ee9b46406c04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:00:59.688780 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.688751    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e89e194-2302-4692-94ca-ee9b46406c04-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:00:59.688780 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:00:59.688776    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t5qp\" (UniqueName: \"kubernetes.io/projected/8e89e194-2302-4692-94ca-ee9b46406c04-kube-api-access-8t5qp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:00.284040 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:00.284011    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"
Nov 01 23:01:00.284543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:00.284046    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz" event={"ID":"8e89e194-2302-4692-94ca-ee9b46406c04","Type":"ContainerDied","Data":"87669f23021d7b6abbdec31dbfe00309f07cf9ba97f50c150ac5af143ae1ddbd"}
Nov 01 23:01:00.284543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:00.284108    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87669f23021d7b6abbdec31dbfe00309f07cf9ba97f50c150ac5af143ae1ddbd"
Nov 01 23:01:00.582976 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:00.582891    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:00.818913 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:00.818875    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" path="/var/lib/kubelet/pods/1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21/volumes"
Nov 01 23:01:01.142684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:01.142650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:02.195691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:02.195666    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:01:05.428503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:05.428459    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:06.117452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:06.117420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:07.199182 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:07.199148    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:07.784083 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:07.784051    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"]
Nov 01 23:01:07.891109 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:07.891079    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"]
Nov 01 23:01:08.219476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:08.219440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:08.423849 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:08.423815    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:09.515750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:09.509644    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"]
Nov 01 23:01:09.515750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:09.509904    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" podUID="45266f72-2dae-4659-9fe5-0c11d5668cc7" containerName="maas-api" containerID="cri-o://2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115" gracePeriod=30
Nov 01 23:01:09.705963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:09.705895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:09.963676 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:09.963647    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 23:01:10.046517 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.046484    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca\") pod \"45266f72-2dae-4659-9fe5-0c11d5668cc7\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") "
Nov 01 23:01:10.046695 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.046616    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs\") pod \"45266f72-2dae-4659-9fe5-0c11d5668cc7\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") "
Nov 01 23:01:10.046785 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.046695    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\") pod \"45266f72-2dae-4659-9fe5-0c11d5668cc7\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") "
Nov 01 23:01:10.046785 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.046764    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc\" (UniqueName: \"kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc\") pod \"45266f72-2dae-4659-9fe5-0c11d5668cc7\" (UID: \"45266f72-2dae-4659-9fe5-0c11d5668cc7\") "
Nov 01 23:01:10.046954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.046931    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca" pod "45266f72-2dae-4659-9fe5-0c11d5668cc7" (UID: "45266f72-2dae-4659-9fe5-0c11d5668cc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:10.047461 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.047434    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45266f72-2dae-4659-9fe5-0c11d5668cc7-openshift-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:10.049024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.048998    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls" pod "45266f72-2dae-4659-9fe5-0c11d5668cc7" (UID: "45266f72-2dae-4659-9fe5-0c11d5668cc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:10.049138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.049087    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs" pod "45266f72-2dae-4659-9fe5-0c11d5668cc7" (UID: "45266f72-2dae-4659-9fe5-0c11d5668cc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:10.049138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.049121    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc" pod "45266f72-2dae-4659-9fe5-0c11d5668cc7" (UID: "45266f72-2dae-4659-9fe5-0c11d5668cc7"). InnerVolumeSpecName "kube-api-access-clftc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:10.149116 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.149083    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:10.149116 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.149111    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45266f72-2dae-4659-9fe5-0c11d5668cc7-maas-api-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:10.149116 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.149121    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clftc\" (UniqueName: \"kubernetes.io/projected/45266f72-2dae-4659-9fe5-0c11d5668cc7-kube-api-access-clftc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:10.379243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.379199    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:10.393208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.393180    4531 generic.go:356] "Generic (PLEG): container finished" podID="45266f72-2dae-4659-9fe5-0c11d5668cc7" containerID="2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115" exitCode=0
Nov 01 23:01:10.393372 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.393223    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"
Nov 01 23:01:10.393372 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.393252    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" event={"ID":"45266f72-2dae-4659-9fe5-0c11d5668cc7","Type":"ContainerDied","Data":"2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115"}
Nov 01 23:01:10.393372 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.393301    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2" event={"ID":"45266f72-2dae-4659-9fe5-0c11d5668cc7","Type":"ContainerDied","Data":"a3c06841c41d195dbd1893d9751b55fd25df50e92774aa27fed2d783ad9dbc01"}
Nov 01 23:01:10.393372 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.393322    4531 scope.go:122] "RemoveContainer" containerID="2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115"
Nov 01 23:01:10.405080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.405060    4531 scope.go:122] "RemoveContainer" containerID="2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115"
Nov 01 23:01:10.405393 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:01:10.405359    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115\": container with ID starting with 2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115 not found: ID does not exist" containerID="2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115"
Nov 01 23:01:10.405445 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.405402    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115"} err="failed to get container status \"2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115\": rpc error: code = NotFound desc = could not find container \"2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115\": container with ID starting with 2216f8599ff324ec6a9ad8f8aeccc2515db10026262f7cb5f385f72039569115 not found: ID does not exist"
Nov 01 23:01:10.553707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.553679    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"]
Nov 01 23:01:10.563121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.563100    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-7da9ed-57cf5f4fd4-zglr2"]
Nov 01 23:01:10.603534 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.603514    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:10.821504 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.821470    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45266f72-2dae-4659-9fe5-0c11d5668cc7" path="/var/lib/kubelet/pods/45266f72-2dae-4659-9fe5-0c11d5668cc7/volumes"
Nov 01 23:01:10.855660 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:10.855630    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:12.528502 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:01:12.528468    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:01:12.571806 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.571780    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:12.736997 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.736965    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"]
Nov 01 23:01:12.764581 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.764531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:12.783550 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.783488    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-bd7ff8-5jr5k"]
Nov 01 23:01:12.785111 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.785079    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5" podUID="49c48b03-dd06-4755-a8d6-f4475978e0cd" containerName="payload-processing" containerID="cri-o://a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d" gracePeriod=25
Nov 01 23:01:12.819357 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.819331    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="538dbe35-3802-4360-857d-4948e966ed90" path="/var/lib/kubelet/pods/538dbe35-3802-4360-857d-4948e966ed90/volumes"
Nov 01 23:01:12.892199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:12.892162    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x" podUID="0acc5535-72ee-4a61-87f1-ac8a28bc836b" containerName="payload-pre-processing" containerID="cri-o://86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb" gracePeriod=25
Nov 01 23:01:13.049817 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.049659    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"]
Nov 01 23:01:13.085811 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.085790    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 23:01:13.099005 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.098976    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05f6bd6c-k9wvm"]
Nov 01 23:01:13.187624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.187598    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r\" (UniqueName: \"kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r\") pod \"49c48b03-dd06-4755-a8d6-f4475978e0cd\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") "
Nov 01 23:01:13.187826 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.187777    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume\") pod \"49c48b03-dd06-4755-a8d6-f4475978e0cd\" (UID: \"49c48b03-dd06-4755-a8d6-f4475978e0cd\") "
Nov 01 23:01:13.188217 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.188187    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume" pod "49c48b03-dd06-4755-a8d6-f4475978e0cd" (UID: "49c48b03-dd06-4755-a8d6-f4475978e0cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:13.189763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.189704    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r" pod "49c48b03-dd06-4755-a8d6-f4475978e0cd" (UID: "49c48b03-dd06-4755-a8d6-f4475978e0cd"). InnerVolumeSpecName "kube-api-access-ktv8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:13.211225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.211196    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 23:01:13.289317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.289291    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume\") pod \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") "
Nov 01 23:01:13.289478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.289395    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2\" (UniqueName: \"kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2\") pod \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\" (UID: \"0acc5535-72ee-4a61-87f1-ac8a28bc836b\") "
Nov 01 23:01:13.289769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.289742    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume" pod "0acc5535-72ee-4a61-87f1-ac8a28bc836b" (UID: "0acc5535-72ee-4a61-87f1-ac8a28bc836b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:13.289952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.289932    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv8r\" (UniqueName: \"kubernetes.io/projected/49c48b03-dd06-4755-a8d6-f4475978e0cd-kube-api-access-ktv8r\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:13.289994 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.289958    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49c48b03-dd06-4755-a8d6-f4475978e0cd-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:13.289994 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.289973    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0acc5535-72ee-4a61-87f1-ac8a28bc836b-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:13.291579 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.291555    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2" pod "0acc5535-72ee-4a61-87f1-ac8a28bc836b" (UID: "0acc5535-72ee-4a61-87f1-ac8a28bc836b"). InnerVolumeSpecName "kube-api-access-tq5h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:13.391691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.391603    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq5h2\" (UniqueName: \"kubernetes.io/projected/0acc5535-72ee-4a61-87f1-ac8a28bc836b-kube-api-access-tq5h2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:13.432920 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.432885    4531 generic.go:356] "Generic (PLEG): container finished" podID="0acc5535-72ee-4a61-87f1-ac8a28bc836b" containerID="86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb" exitCode=0
Nov 01 23:01:13.433110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.432956    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"
Nov 01 23:01:13.433194 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.433112    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x" event={"ID":"0acc5535-72ee-4a61-87f1-ac8a28bc836b","Type":"ContainerDied","Data":"86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb"}
Nov 01 23:01:13.433402 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.433189    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x" event={"ID":"0acc5535-72ee-4a61-87f1-ac8a28bc836b","Type":"ContainerDied","Data":"a3cee84529900a2db301e13b49b214e8d78a3f6dfc240a5d283b78393ae83a7a"}
Nov 01 23:01:13.433402 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.433227    4531 scope.go:122] "RemoveContainer" containerID="86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb"
Nov 01 23:01:13.435648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.435625    4531 generic.go:356] "Generic (PLEG): container finished" podID="49c48b03-dd06-4755-a8d6-f4475978e0cd" containerID="a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d" exitCode=0
Nov 01 23:01:13.435803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.435668    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5" event={"ID":"49c48b03-dd06-4755-a8d6-f4475978e0cd","Type":"ContainerDied","Data":"a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d"}
Nov 01 23:01:13.435803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.435682    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"
Nov 01 23:01:13.435803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.435701    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5" event={"ID":"49c48b03-dd06-4755-a8d6-f4475978e0cd","Type":"ContainerDied","Data":"d351ae3b34c07f7cfe0b7bafbed45a1d5b5fad4c3e4ef42566ca66338fadd884"}
Nov 01 23:01:13.447450 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.447430    4531 scope.go:122] "RemoveContainer" containerID="86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb"
Nov 01 23:01:13.447736 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:01:13.447706    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb\": container with ID starting with 86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb not found: ID does not exist" containerID="86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb"
Nov 01 23:01:13.447811 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.447760    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb"} err="failed to get container status \"86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb\": rpc error: code = NotFound desc = could not find container \"86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb\": container with ID starting with 86284ed71c72c14580ef30746b3afaddfc4efb70c1fb3cc2d2eeff726fec3dfb not found: ID does not exist"
Nov 01 23:01:13.447811 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.447779    4531 scope.go:122] "RemoveContainer" containerID="a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d"
Nov 01 23:01:13.457432 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.457418    4531 scope.go:122] "RemoveContainer" containerID="a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d"
Nov 01 23:01:13.457758 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:01:13.457696    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d\": container with ID starting with a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d not found: ID does not exist" containerID="a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d"
Nov 01 23:01:13.457843 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.457761    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d"} err="failed to get container status \"a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d\": rpc error: code = NotFound desc = could not find container \"a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d\": container with ID starting with a623356273742190950685e0adafcc02d0e261f472981f3c3b2dda6c5cce0d4d not found: ID does not exist"
Nov 01 23:01:13.592486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.592451    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"]
Nov 01 23:01:13.601394 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.601365    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7da9ed-64d9b6479-n448x"]
Nov 01 23:01:13.703988 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.703952    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"]
Nov 01 23:01:13.710597 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.710574    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-7da9ed-55767f74bf-hftd5"]
Nov 01 23:01:13.739994 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.739965    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:13.949721 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:13.949684    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:14.817928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:14.817898    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acc5535-72ee-4a61-87f1-ac8a28bc836b" path="/var/lib/kubelet/pods/0acc5535-72ee-4a61-87f1-ac8a28bc836b/volumes"
Nov 01 23:01:14.818476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:14.818455    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49c48b03-dd06-4755-a8d6-f4475978e0cd" path="/var/lib/kubelet/pods/49c48b03-dd06-4755-a8d6-f4475978e0cd/volumes"
Nov 01 23:01:14.818977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:14.818948    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d65dd531-32d4-4276-960d-471d56d531ee" path="/var/lib/kubelet/pods/d65dd531-32d4-4276-960d-471d56d531ee/volumes"
Nov 01 23:01:17.732858 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:17.732826    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:18.163130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:18.163043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:18.425279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:18.425202    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:18.713425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:18.713396    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:19.299279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299247    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"]
Nov 01 23:01:19.299895 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299879    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerName="payload-pre-processing"
Nov 01 23:01:19.299953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299896    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerName="payload-pre-processing"
Nov 01 23:01:19.299953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299927    4531 cpu_manager.go:414] "containerMap: removing container" podUID="45266f72-2dae-4659-9fe5-0c11d5668cc7" containerName="maas-api"
Nov 01 23:01:19.299953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299932    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45266f72-2dae-4659-9fe5-0c11d5668cc7" containerName="maas-api"
Nov 01 23:01:19.299953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299942    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0acc5535-72ee-4a61-87f1-ac8a28bc836b" containerName="payload-pre-processing"
Nov 01 23:01:19.299953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299947    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acc5535-72ee-4a61-87f1-ac8a28bc836b" containerName="payload-pre-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299956    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerName="payload-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299960    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerName="payload-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299975    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8e89e194-2302-4692-94ca-ee9b46406c04" containerName="revoke-keys"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299980    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e89e194-2302-4692-94ca-ee9b46406c04" containerName="revoke-keys"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299990    4531 cpu_manager.go:414] "containerMap: removing container" podUID="49c48b03-dd06-4755-a8d6-f4475978e0cd" containerName="payload-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.299995    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49c48b03-dd06-4755-a8d6-f4475978e0cd" containerName="payload-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.300086    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e89e194-2302-4692-94ca-ee9b46406c04" containerName="revoke-keys"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.300095    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="49c48b03-dd06-4755-a8d6-f4475978e0cd" containerName="payload-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.300103    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac9bf77-abe7-4fd5-b7d5-7b1d0e406e21" containerName="payload-pre-processing"
Nov 01 23:01:19.300110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.300114    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="45266f72-2dae-4659-9fe5-0c11d5668cc7" containerName="maas-api"
Nov 01 23:01:19.300409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.300120    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acc5535-72ee-4a61-87f1-ac8a28bc836b" containerName="payload-pre-processing"
Nov 01 23:01:19.300409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.300144    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f33ff84c-4c87-4ec2-9a33-25cef07051a0" containerName="payload-processing"
Nov 01 23:01:19.305567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.305543    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.358572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.357886    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"]
Nov 01 23:01:19.366307 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.366283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.366580 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.366562    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctknc\" (UniqueName: \"kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.366981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.366955    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.398825 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.398793    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:01:19.398937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.398858    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:01:19.468196 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.468168    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.468338 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.468252    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.468338 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.468291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctknc\" (UniqueName: \"kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.469017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.468991    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.471054 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.471031    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.522477 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.522455    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctknc\" (UniqueName: \"kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc\") pod \"maas-api-revoke-keys-e2e-api-b-584561-7ljbg\" (UID: \"98295795-d339-4700-a606-3671398b2341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:19.620321 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:19.620256    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:20.000967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:20.000937    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"]
Nov 01 23:01:20.004898 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:01:20.004867    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98295795_d339_4700_a606_3671398b2341.slice/crio-ecb0e4eee28d6022802c6afef68696c9d500d2818574cdcf762e5735011ef35e WatchSource:0}: Error finding container ecb0e4eee28d6022802c6afef68696c9d500d2818574cdcf762e5735011ef35e: Status 404 returned error can't find the container with id ecb0e4eee28d6022802c6afef68696c9d500d2818574cdcf762e5735011ef35e
Nov 01 23:01:20.509112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:20.507774    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg" event={"ID":"98295795-d339-4700-a606-3671398b2341","Type":"ContainerStarted","Data":"b1e6b3997eb7c51a73df6546e16c470584f44c0c24d7c14b6ac56fff64280ef8"}
Nov 01 23:01:20.509112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:20.507844    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg" event={"ID":"98295795-d339-4700-a606-3671398b2341","Type":"ContainerStarted","Data":"ecb0e4eee28d6022802c6afef68696c9d500d2818574cdcf762e5735011ef35e"}
Nov 01 23:01:20.619708 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:20.619597    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg" podStartSLOduration=1.6195780530000001 podStartE2EDuration="1.619578053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:20.616874339 +0000 UTC m=+2261.934524339" watchObservedRunningTime="2026-11-01 23:01:20.619578053 +0000 UTC m=+2261.937228057"
Nov 01 23:01:21.520256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:21.520220    4531 generic.go:356] "Generic (PLEG): container finished" podID="98295795-d339-4700-a606-3671398b2341" containerID="b1e6b3997eb7c51a73df6546e16c470584f44c0c24d7c14b6ac56fff64280ef8" exitCode=0
Nov 01 23:01:21.520657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:21.520308    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg" event={"ID":"98295795-d339-4700-a606-3671398b2341","Type":"ContainerDied","Data":"b1e6b3997eb7c51a73df6546e16c470584f44c0c24d7c14b6ac56fff64280ef8"}
Nov 01 23:01:22.452027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.451990    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:22.643539 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.643503    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:22.693406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.693382    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:22.814425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.814352    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc\" (UniqueName: \"kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc\") pod \"98295795-d339-4700-a606-3671398b2341\" (UID: \"98295795-d339-4700-a606-3671398b2341\") "
Nov 01 23:01:22.814586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.814478    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca\") pod \"98295795-d339-4700-a606-3671398b2341\" (UID: \"98295795-d339-4700-a606-3671398b2341\") "
Nov 01 23:01:22.814586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.814558    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca\") pod \"98295795-d339-4700-a606-3671398b2341\" (UID: \"98295795-d339-4700-a606-3671398b2341\") "
Nov 01 23:01:22.815041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.814849    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca" pod "98295795-d339-4700-a606-3671398b2341" (UID: "98295795-d339-4700-a606-3671398b2341"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:22.815160 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.815096    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98295795-d339-4700-a606-3671398b2341-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:22.817475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.817170    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc" pod "98295795-d339-4700-a606-3671398b2341" (UID: "98295795-d339-4700-a606-3671398b2341"). InnerVolumeSpecName "kube-api-access-ctknc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:22.817475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.817394    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca" pod "98295795-d339-4700-a606-3671398b2341" (UID: "98295795-d339-4700-a606-3671398b2341"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:22.916844 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.916809    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctknc\" (UniqueName: \"kubernetes.io/projected/98295795-d339-4700-a606-3671398b2341-kube-api-access-ctknc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:22.916844 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:22.916839    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98295795-d339-4700-a606-3671398b2341-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:23.144582 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:23.144490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:23.539939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:23.539904    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"
Nov 01 23:01:23.540145 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:23.539903    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg" event={"ID":"98295795-d339-4700-a606-3671398b2341","Type":"ContainerDied","Data":"ecb0e4eee28d6022802c6afef68696c9d500d2818574cdcf762e5735011ef35e"}
Nov 01 23:01:23.540145 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:23.540036    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecb0e4eee28d6022802c6afef68696c9d500d2818574cdcf762e5735011ef35e"
Nov 01 23:01:28.226196 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:28.226164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:31.322100 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:31.322061    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:31.509719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:31.509678    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:33.047505 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:33.047379    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:37.930065 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:37.930023    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:40.568181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:40.568114    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:42.528020 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:01:42.527979    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:01:42.843142 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:42.843048    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:46.826695 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:46.826659    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"]
Nov 01 23:01:46.827165 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:46.826882    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk" podUID="0a65c30d-fd4d-4540-a8fc-f7e83fa24142" containerName="maas-api" containerID="cri-o://16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9" gracePeriod=30
Nov 01 23:01:46.989186 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:46.989155    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:47.146567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.146534    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 23:01:47.242616 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.242576    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs\") pod \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") "
Nov 01 23:01:47.242822 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.242678    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2\" (UniqueName: \"kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2\") pod \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") "
Nov 01 23:01:47.242822 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.242753    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls\") pod \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") "
Nov 01 23:01:47.242947 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.242864    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca\") pod \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\" (UID: \"0a65c30d-fd4d-4540-a8fc-f7e83fa24142\") "
Nov 01 23:01:47.244102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.244024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca" pod "0a65c30d-fd4d-4540-a8fc-f7e83fa24142" (UID: "0a65c30d-fd4d-4540-a8fc-f7e83fa24142"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:47.247033 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.247009    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2" pod "0a65c30d-fd4d-4540-a8fc-f7e83fa24142" (UID: "0a65c30d-fd4d-4540-a8fc-f7e83fa24142"). InnerVolumeSpecName "kube-api-access-qhwc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:47.252486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.248780    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls" pod "0a65c30d-fd4d-4540-a8fc-f7e83fa24142" (UID: "0a65c30d-fd4d-4540-a8fc-f7e83fa24142"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:47.262773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.257210    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs" pod "0a65c30d-fd4d-4540-a8fc-f7e83fa24142" (UID: "0a65c30d-fd4d-4540-a8fc-f7e83fa24142"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:47.344533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.344498    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:47.344533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.344526    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhwc2\" (UniqueName: \"kubernetes.io/projected/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-kube-api-access-qhwc2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:47.344533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.344536    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-maas-api-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:47.344804 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.344546    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a65c30d-fd4d-4540-a8fc-f7e83fa24142-openshift-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:47.748035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.747996    4531 generic.go:356] "Generic (PLEG): container finished" podID="0a65c30d-fd4d-4540-a8fc-f7e83fa24142" containerID="16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9" exitCode=0
Nov 01 23:01:47.748215 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.748051    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"
Nov 01 23:01:47.748215 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.748084    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk" event={"ID":"0a65c30d-fd4d-4540-a8fc-f7e83fa24142","Type":"ContainerDied","Data":"16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9"}
Nov 01 23:01:47.748215 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.748163    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk" event={"ID":"0a65c30d-fd4d-4540-a8fc-f7e83fa24142","Type":"ContainerDied","Data":"ee5370b2aa2bb1cb4f43b9f5e45823caaca0bda920dea01d43152600b6246a94"}
Nov 01 23:01:47.748215 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.748192    4531 scope.go:122] "RemoveContainer" containerID="16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9"
Nov 01 23:01:47.779582 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.768248    4531 scope.go:122] "RemoveContainer" containerID="16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9"
Nov 01 23:01:47.779582 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:01:47.769475    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9\": container with ID starting with 16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9 not found: ID does not exist" containerID="16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9"
Nov 01 23:01:47.779582 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.769567    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9"} err="failed to get container status \"16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9\": rpc error: code = NotFound desc = could not find container \"16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9\": container with ID starting with 16129c05803265c44ca43c98ca60657e3234463df4ba3add5044a28fb56060a9 not found: ID does not exist"
Nov 01 23:01:47.908831 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.908798    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"]
Nov 01 23:01:47.954395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:47.954363    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-103369-6ddccb6749-7hrpk"]
Nov 01 23:01:48.413659 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.413628    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"]
Nov 01 23:01:48.414264 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.414250    4531 cpu_manager.go:414] "containerMap: removing container" podUID="98295795-d339-4700-a606-3671398b2341" containerName="revoke-keys"
Nov 01 23:01:48.414310 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.414265    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98295795-d339-4700-a606-3671398b2341" containerName="revoke-keys"
Nov 01 23:01:48.414348 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.414310    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0a65c30d-fd4d-4540-a8fc-f7e83fa24142" containerName="maas-api"
Nov 01 23:01:48.414348 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.414316    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a65c30d-fd4d-4540-a8fc-f7e83fa24142" containerName="maas-api"
Nov 01 23:01:48.414423 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.414414    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a65c30d-fd4d-4540-a8fc-f7e83fa24142" containerName="maas-api"
Nov 01 23:01:48.414460 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.414426    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="98295795-d339-4700-a606-3671398b2341" containerName="revoke-keys"
Nov 01 23:01:48.419331 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.419309    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.470154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.470134    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:01:48.470154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.470147    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:01:48.513685 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.513652    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"]
Nov 01 23:01:48.556449 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.556422    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:48.564043 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.564014    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crhmm\" (UniqueName: \"kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.564241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.564218    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.564422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.564398    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.665846 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.665578    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.665846 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.665696    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crhmm\" (UniqueName: \"kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.666405 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.665866    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.666960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.666864    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.678566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.678529    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.724297 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.724268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crhmm\" (UniqueName: \"kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm\") pod \"maas-api-revoke-sub-8617f7ac-js45h\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.733213 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.733189    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:48.753766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.753697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:48.820066 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.820032    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a65c30d-fd4d-4540-a8fc-f7e83fa24142" path="/var/lib/kubelet/pods/0a65c30d-fd4d-4540-a8fc-f7e83fa24142/volumes"
Nov 01 23:01:48.948105 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.948075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:48.995666 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.995640    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"]
Nov 01 23:01:48.999584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:48.999559    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-103369-bwt2w"]
Nov 01 23:01:49.108203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:49.108179    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"]
Nov 01 23:01:49.111948 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:01:49.111925    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc66b6f_a6a3_499b_98ea_e91ab6ef9cfe.slice/crio-3b9bbfec9086c10c91a48de72dd0c6d050ac52f78e4ac4875bc64580a6648bca WatchSource:0}: Error finding container 3b9bbfec9086c10c91a48de72dd0c6d050ac52f78e4ac4875bc64580a6648bca: Status 404 returned error can't find the container with id 3b9bbfec9086c10c91a48de72dd0c6d050ac52f78e4ac4875bc64580a6648bca
Nov 01 23:01:49.428129 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:49.428069    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:49.767897 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:49.767858    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h" event={"ID":"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe","Type":"ContainerStarted","Data":"42391f02ce417ce978dca0cdadef0ae04db51d79c0b4c5d3fede71cdf8f2f3fb"}
Nov 01 23:01:49.768063 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:49.767922    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h" event={"ID":"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe","Type":"ContainerStarted","Data":"3b9bbfec9086c10c91a48de72dd0c6d050ac52f78e4ac4875bc64580a6648bca"}
Nov 01 23:01:49.893923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:49.893871    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h" podStartSLOduration=1.8938527010000001 podStartE2EDuration="1.893852701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:01:49.890911138 +0000 UTC m=+2291.208561139" watchObservedRunningTime="2026-11-01 23:01:49.893852701 +0000 UTC m=+2291.211502705"
Nov 01 23:01:50.777085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:50.777053    4531 generic.go:356] "Generic (PLEG): container finished" podID="fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" containerID="42391f02ce417ce978dca0cdadef0ae04db51d79c0b4c5d3fede71cdf8f2f3fb" exitCode=0
Nov 01 23:01:50.777515 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:50.777103    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h" event={"ID":"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe","Type":"ContainerDied","Data":"42391f02ce417ce978dca0cdadef0ae04db51d79c0b4c5d3fede71cdf8f2f3fb"}
Nov 01 23:01:50.819093 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:50.819065    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bc9e97-890c-45f5-b66d-5122e9dd9fa7" path="/var/lib/kubelet/pods/90bc9e97-890c-45f5-b66d-5122e9dd9fa7/volumes"
Nov 01 23:01:51.967477 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:51.967440    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:52.017651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.017619    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm\" (UniqueName: \"kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm\") pod \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") "
Nov 01 23:01:52.017900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.017881    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca\") pod \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") "
Nov 01 23:01:52.017950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.017938    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca\") pod \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\" (UID: \"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe\") "
Nov 01 23:01:52.018301 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.018274    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca" pod "fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" (UID: "fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:01:52.018409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.018397    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:52.019723 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.019698    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm" pod "fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" (UID: "fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe"). InnerVolumeSpecName "kube-api-access-crhmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:01:52.019918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.019902    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca" pod "fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" (UID: "fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:01:52.120143 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.120071    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:52.120143 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.120098    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crhmm\" (UniqueName: \"kubernetes.io/projected/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe-kube-api-access-crhmm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:01:52.794468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.794438    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"
Nov 01 23:01:52.794468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.794460    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h" event={"ID":"fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe","Type":"ContainerDied","Data":"3b9bbfec9086c10c91a48de72dd0c6d050ac52f78e4ac4875bc64580a6648bca"}
Nov 01 23:01:52.794701 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:52.794525    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b9bbfec9086c10c91a48de72dd0c6d050ac52f78e4ac4875bc64580a6648bca"
Nov 01 23:01:54.175084 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:54.175018    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:55.302090 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:55.302053    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:55.518526 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:55.518493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:55.721750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:55.720583    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:56.438179 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:56.438145    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:57.035780 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:57.035616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:57.446342 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:57.446292    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:59.164478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.164442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:59.371950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.371912    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:01:59.655768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.655735    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"]
Nov 01 23:01:59.656412 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.656397    4531 cpu_manager.go:414] "containerMap: removing container" podUID="fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" containerName="revoke-keys"
Nov 01 23:01:59.656459 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.656414    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" containerName="revoke-keys"
Nov 01 23:01:59.656550 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.656539    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" containerName="revoke-keys"
Nov 01 23:01:59.660478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.660451    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.735907 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.735875    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.736088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.735920    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.736088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.735947    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qs7s\" (UniqueName: \"kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.756468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.756439    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:01:59.756673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.756557    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:01:59.763505 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.763478    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"]
Nov 01 23:01:59.837298 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.837251    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.837456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.837300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.837456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.837331    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qs7s\" (UniqueName: \"kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.838582 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.838552    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.840075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.840047    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.891636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.891611    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qs7s\" (UniqueName: \"kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s\") pod \"maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:01:59.976024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:01:59.975999    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:02:00.360791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:00.360649    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"]
Nov 01 23:02:00.364168 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:02:00.364128    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1139fab9_c1ee_4e3e_aa72_a07e4d49cdf2.slice/crio-97ffdfe32b1820bfc329a5f98fed41591820e0f190c516bbcaccab76c47d944c WatchSource:0}: Error finding container 97ffdfe32b1820bfc329a5f98fed41591820e0f190c516bbcaccab76c47d944c: Status 404 returned error can't find the container with id 97ffdfe32b1820bfc329a5f98fed41591820e0f190c516bbcaccab76c47d944c
Nov 01 23:02:00.866073 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:00.866027    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c" event={"ID":"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2","Type":"ContainerStarted","Data":"b0b231fdcae44bbe4f53f0abca1955ea39e9111d00590e5f8c6248e2682e40a3"}
Nov 01 23:02:00.866249 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:00.866120    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c" event={"ID":"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2","Type":"ContainerStarted","Data":"97ffdfe32b1820bfc329a5f98fed41591820e0f190c516bbcaccab76c47d944c"}
Nov 01 23:02:00.980243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:00.980189    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c" podStartSLOduration=1.980172912 podStartE2EDuration="1.980172912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:01:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:00.973893982 +0000 UTC m=+2302.291543988" watchObservedRunningTime="2026-11-01 23:02:00.980172912 +0000 UTC m=+2302.297822915"
Nov 01 23:02:01.606628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:01.606598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:01.832076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:01.832045    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:01.876205 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:01.876171    4531 generic.go:356] "Generic (PLEG): container finished" podID="1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" containerID="b0b231fdcae44bbe4f53f0abca1955ea39e9111d00590e5f8c6248e2682e40a3" exitCode=0
Nov 01 23:02:01.876380 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:01.876220    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c" event={"ID":"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2","Type":"ContainerDied","Data":"b0b231fdcae44bbe4f53f0abca1955ea39e9111d00590e5f8c6248e2682e40a3"}
Nov 01 23:02:03.046675 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.046651    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:02:03.182080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.181988    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca\") pod \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") "
Nov 01 23:02:03.182080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.182040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca\") pod \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") "
Nov 01 23:02:03.182333 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.182145    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s\" (UniqueName: \"kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s\") pod \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\" (UID: \"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2\") "
Nov 01 23:02:03.182926 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.182749    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca" pod "1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" (UID: "1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:03.182926 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.182892    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:03.184405 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.184373    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca" pod "1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" (UID: "1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:03.184863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.184838    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s" pod "1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" (UID: "1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2"). InnerVolumeSpecName "kube-api-access-8qs7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:03.283943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.283909    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:03.283943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.283943    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qs7s\" (UniqueName: \"kubernetes.io/projected/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2-kube-api-access-8qs7s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:03.896096 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.896069    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"
Nov 01 23:02:03.896295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.896101    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c" event={"ID":"1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2","Type":"ContainerDied","Data":"97ffdfe32b1820bfc329a5f98fed41591820e0f190c516bbcaccab76c47d944c"}
Nov 01 23:02:03.896295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:03.896150    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ffdfe32b1820bfc329a5f98fed41591820e0f190c516bbcaccab76c47d944c"
Nov 01 23:02:04.063642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:04.063609    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"]
Nov 01 23:02:04.076205 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:04.076170    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"]
Nov 01 23:02:04.271078 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:04.271041    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:04.482610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:04.482577    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:04.705966 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:04.705933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:04.911750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:04.902990    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:05.081823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.081720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:05.228164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.228133    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"]
Nov 01 23:02:05.228882 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.228865    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" containerName="revoke-keys"
Nov 01 23:02:05.228938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.228884    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" containerName="revoke-keys"
Nov 01 23:02:05.229008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.228998    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" containerName="revoke-keys"
Nov 01 23:02:05.232521 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.232489    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.282285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.282258    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-55b58e-dockercfg-hrt46\""
Nov 01 23:02:05.282466 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.282371    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-55b58e\""
Nov 01 23:02:05.286546 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.286527    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"]
Nov 01 23:02:05.424512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.414450    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.424512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.414534    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnmdc\" (UniqueName: \"kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc\") pod \"payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.436587 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.435808    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"]
Nov 01 23:02:05.441604 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.441570    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.451264 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.451229    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"]
Nov 01 23:02:05.516041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.516006    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.516241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.516224    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.516311 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.516300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnmdc\" (UniqueName: \"kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc\") pod \"payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.516366 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.516337    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jttld\" (UniqueName: \"kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld\") pod \"payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.517050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.517024    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.569930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.569895    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnmdc\" (UniqueName: \"kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc\") pod \"payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:05.617531 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.617505    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jttld\" (UniqueName: \"kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld\") pod \"payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.617677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.617612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.618314 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.618296    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.670837 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.670801    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jttld\" (UniqueName: \"kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld\") pod \"payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") " pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.772889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.772860    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:05.850976 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:05.850937    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:06.151008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.150921    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"]
Nov 01 23:02:06.154351 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:02:06.154321    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718b599c_fc6e_4a5b_a91b_abcefda5a3fc.slice/crio-db289fbe0b42ee5d78b5e121febbd57688f5b39b94ca7913c05c1e2a23576f15 WatchSource:0}: Error finding container db289fbe0b42ee5d78b5e121febbd57688f5b39b94ca7913c05c1e2a23576f15: Status 404 returned error can't find the container with id db289fbe0b42ee5d78b5e121febbd57688f5b39b94ca7913c05c1e2a23576f15
Nov 01 23:02:06.253123 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.253100    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"]
Nov 01 23:02:06.256319 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:02:06.256294    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ff5454_8242_403b_893e_56a69caaafa2.slice/crio-2f0da0459e692df83ae54e6a3ef27901c009b116acf189febad56ba5e979ef00 WatchSource:0}: Error finding container 2f0da0459e692df83ae54e6a3ef27901c009b116acf189febad56ba5e979ef00: Status 404 returned error can't find the container with id 2f0da0459e692df83ae54e6a3ef27901c009b116acf189febad56ba5e979ef00
Nov 01 23:02:06.957901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.957865    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d" event={"ID":"68ff5454-8242-403b-893e-56a69caaafa2","Type":"ContainerStarted","Data":"d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb"}
Nov 01 23:02:06.958117 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.957928    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d" event={"ID":"68ff5454-8242-403b-893e-56a69caaafa2","Type":"ContainerStarted","Data":"2f0da0459e692df83ae54e6a3ef27901c009b116acf189febad56ba5e979ef00"}
Nov 01 23:02:06.958117 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.958069    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:06.959628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.959606    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h" event={"ID":"718b599c-fc6e-4a5b-a91b-abcefda5a3fc","Type":"ContainerStarted","Data":"b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c"}
Nov 01 23:02:06.959755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.959641    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h" event={"ID":"718b599c-fc6e-4a5b-a91b-abcefda5a3fc","Type":"ContainerStarted","Data":"db289fbe0b42ee5d78b5e121febbd57688f5b39b94ca7913c05c1e2a23576f15"}
Nov 01 23:02:06.959755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:06.959672    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:07.065890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:07.065835    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d" podStartSLOduration=2.065822311 podStartE2EDuration="2.065822311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:07.065106327 +0000 UTC m=+2308.382756330" watchObservedRunningTime="2026-11-01 23:02:07.065822311 +0000 UTC m=+2308.383472348"
Nov 01 23:02:07.181916 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:07.181226    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h" podStartSLOduration=2.181208387 podStartE2EDuration="2.181208387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:07.174708229 +0000 UTC m=+2308.492358231" watchObservedRunningTime="2026-11-01 23:02:07.181208387 +0000 UTC m=+2308.498858388"
Nov 01 23:02:09.451754 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:09.451095    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:10.755007 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:10.754973    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:10.799396 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:10.799366    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"]
Nov 01 23:02:10.902755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:10.902693    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"]
Nov 01 23:02:10.983424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:10.983397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:11.173344 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:11.173233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:11.372713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:11.372676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:11.943423 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:11.943393    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:12.562045 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:12.562008    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:02:15.802843 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:15.802793    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" containerName="payload-processing" containerID="cri-o://de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e" gracePeriod=25
Nov 01 23:02:15.906823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:15.906780    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerName="payload-pre-processing" containerID="cri-o://18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9" gracePeriod=25
Nov 01 23:02:16.791753 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:16.790550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:17.973028 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:17.972998    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:02:17.973404 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:17.973062    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:02:18.378908 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:18.378819    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:18.616231 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:18.615929    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:18.621560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:18.621063    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"]
Nov 01 23:02:18.621560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:18.621306    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy" containerID="cri-o://a5f3a34e0d53fafe5ce608b8db3f5fa503da8c6a610d460fa7110f0139f37c59" gracePeriod=30
Nov 01 23:02:18.874666 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:18.874568    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:19.085899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.085854    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main" containerID="cri-o://05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde" gracePeriod=45
Nov 01 23:02:19.109444 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.109406    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main" containerID="cri-o://21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d" gracePeriod=45
Nov 01 23:02:19.409703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.409310    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 23:02:19.426248 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.426226    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 23:02:19.509315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.509264    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home\") pod \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") "
Nov 01 23:02:19.509503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.509377    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache\") pod \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") "
Nov 01 23:02:19.509503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.509425    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh\" (UniqueName: \"kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh\") pod \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") "
Nov 01 23:02:19.509503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.509493    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs\") pod \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.509962    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache\") pod \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510013    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm\") pod \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510079    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm\") pod \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510114    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir\") pod \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510153    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir\") pod \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510201    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home\") pod \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\" (UID: \"7d9d7fb2-7701-49f7-9184-7d85caba1ed1\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510255    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx\" (UniqueName: \"kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx\") pod \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.510293    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs\" (UniqueName: \"kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs\") pod \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\" (UID: \"996218de-84d9-4c3f-a3b5-4a09e7109fe5\") "
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.512535    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir" pod "7d9d7fb2-7701-49f7-9184-7d85caba1ed1" (UID: "7d9d7fb2-7701-49f7-9184-7d85caba1ed1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.512980    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir" pod "996218de-84d9-4c3f-a3b5-4a09e7109fe5" (UID: "996218de-84d9-4c3f-a3b5-4a09e7109fe5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.513162    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home" pod "7d9d7fb2-7701-49f7-9184-7d85caba1ed1" (UID: "7d9d7fb2-7701-49f7-9184-7d85caba1ed1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.513304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm" pod "996218de-84d9-4c3f-a3b5-4a09e7109fe5" (UID: "996218de-84d9-4c3f-a3b5-4a09e7109fe5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.513534    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache" pod "996218de-84d9-4c3f-a3b5-4a09e7109fe5" (UID: "996218de-84d9-4c3f-a3b5-4a09e7109fe5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.513717    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs" pod "996218de-84d9-4c3f-a3b5-4a09e7109fe5" (UID: "996218de-84d9-4c3f-a3b5-4a09e7109fe5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:19.513932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.513762    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home" pod "996218de-84d9-4c3f-a3b5-4a09e7109fe5" (UID: "996218de-84d9-4c3f-a3b5-4a09e7109fe5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.514738 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.513958    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache" pod "7d9d7fb2-7701-49f7-9184-7d85caba1ed1" (UID: "7d9d7fb2-7701-49f7-9184-7d85caba1ed1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.515719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.515690    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm" pod "7d9d7fb2-7701-49f7-9184-7d85caba1ed1" (UID: "7d9d7fb2-7701-49f7-9184-7d85caba1ed1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:19.517055 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.517034    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh" pod "7d9d7fb2-7701-49f7-9184-7d85caba1ed1" (UID: "7d9d7fb2-7701-49f7-9184-7d85caba1ed1"). InnerVolumeSpecName "kube-api-access-8kpsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:19.528577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.525250    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs" pod "7d9d7fb2-7701-49f7-9184-7d85caba1ed1" (UID: "7d9d7fb2-7701-49f7-9184-7d85caba1ed1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:19.528577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.525262    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx" pod "996218de-84d9-4c3f-a3b5-4a09e7109fe5" (UID: "996218de-84d9-4c3f-a3b5-4a09e7109fe5"). InnerVolumeSpecName "kube-api-access-jm4xx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:19.612427 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612393    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612427 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612422    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kpsh\" (UniqueName: \"kubernetes.io/projected/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-kube-api-access-8kpsh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612427 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612433    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612442    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612452    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612460    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612468    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612475    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612483    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7d9d7fb2-7701-49f7-9184-7d85caba1ed1-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612491    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm4xx\" (UniqueName: \"kubernetes.io/projected/996218de-84d9-4c3f-a3b5-4a09e7109fe5-kube-api-access-jm4xx\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612499    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/996218de-84d9-4c3f-a3b5-4a09e7109fe5-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:19.612684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:19.612506    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/996218de-84d9-4c3f-a3b5-4a09e7109fe5-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:20.125557 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.125523    4531 generic.go:356] "Generic (PLEG): container finished" podID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerID="05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde" exitCode=0
Nov 01 23:02:20.125950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.125571    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"
Nov 01 23:02:20.125950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.125608    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" event={"ID":"996218de-84d9-4c3f-a3b5-4a09e7109fe5","Type":"ContainerDied","Data":"05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde"}
Nov 01 23:02:20.125950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.125653    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r" event={"ID":"996218de-84d9-4c3f-a3b5-4a09e7109fe5","Type":"ContainerDied","Data":"60e56c78d8b1c9d51d5dec833514423a2ad67a533c03eb36b875bf7362be7519"}
Nov 01 23:02:20.125950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.125671    4531 scope.go:122] "RemoveContainer" containerID="05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde"
Nov 01 23:02:20.127345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.127326    4531 generic.go:356] "Generic (PLEG): container finished" podID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerID="21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d" exitCode=0
Nov 01 23:02:20.127458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.127371    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"
Nov 01 23:02:20.127458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.127391    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" event={"ID":"7d9d7fb2-7701-49f7-9184-7d85caba1ed1","Type":"ContainerDied","Data":"21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d"}
Nov 01 23:02:20.127458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.127432    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875" event={"ID":"7d9d7fb2-7701-49f7-9184-7d85caba1ed1","Type":"ContainerDied","Data":"d41e78100fc7d3ba8de471c135923149058034c489fe886918bf39d13d35b4fa"}
Nov 01 23:02:20.137809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.137792    4531 scope.go:122] "RemoveContainer" containerID="05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde"
Nov 01 23:02:20.138065 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:20.138050    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde\": container with ID starting with 05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde not found: ID does not exist" containerID="05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde"
Nov 01 23:02:20.138103 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.138074    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde"} err="failed to get container status \"05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde\": rpc error: code = NotFound desc = could not find container \"05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde\": container with ID starting with 05e41d3bf041c7990e560de26d72261d95e461d842d0134ffa8c8270d9e4ebde not found: ID does not exist"
Nov 01 23:02:20.138103 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.138088    4531 scope.go:122] "RemoveContainer" containerID="21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d"
Nov 01 23:02:20.148079 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.148058    4531 scope.go:122] "RemoveContainer" containerID="21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d"
Nov 01 23:02:20.148334 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:20.148318    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d\": container with ID starting with 21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d not found: ID does not exist" containerID="21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d"
Nov 01 23:02:20.148382 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.148339    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d"} err="failed to get container status \"21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d\": rpc error: code = NotFound desc = could not find container \"21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d\": container with ID starting with 21790f8d28dc3a424da0500806939c737b6972578f83b2e559f870c8ae062a3d not found: ID does not exist"
Nov 01 23:02:20.282526 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.282498    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"]
Nov 01 23:02:20.286468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.286443    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/distinct-w5-d6101b-kserve-c946db95b-r9m8r"]
Nov 01 23:02:20.430873 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.430811    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"]
Nov 01 23:02:20.434626 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.434605    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-d6101b/premium-w5-d6101b-kserve-54b97dd857-28875"]
Nov 01 23:02:20.820437 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.820407    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" path="/var/lib/kubelet/pods/7d9d7fb2-7701-49f7-9184-7d85caba1ed1/volumes"
Nov 01 23:02:20.821009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:20.820994    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" path="/var/lib/kubelet/pods/996218de-84d9-4c3f-a3b5-4a09e7109fe5/volumes"
Nov 01 23:02:21.262607 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:21.262571    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:02:21.262607 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:21.262595    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:02:21.954499 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:21.954441    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:24.164945 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.164913    4531 generic.go:356] "Generic (PLEG): container finished" podID="30ea652e-3239-40dd-9c46-c826105e70ed" containerID="a5f3a34e0d53fafe5ce608b8db3f5fa503da8c6a610d460fa7110f0139f37c59" exitCode=0
Nov 01 23:02:24.165371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.164995    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" event={"ID":"30ea652e-3239-40dd-9c46-c826105e70ed","Type":"ContainerDied","Data":"a5f3a34e0d53fafe5ce608b8db3f5fa503da8c6a610d460fa7110f0139f37c59"}
Nov 01 23:02:24.316369 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.316346    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 23:02:24.473842 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.473806    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.474024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.473848    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.474024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.473930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.474360 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.474332    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.474597 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.474365    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.474597 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.474437    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.475056 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.474989    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.475183 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.475167    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.475334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.475319    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.475482 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.475467    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.475589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.475576    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.475855 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.475785    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy\") pod \"30ea652e-3239-40dd-9c46-c826105e70ed\" (UID: \"30ea652e-3239-40dd-9c46-c826105e70ed\") "
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.476588    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.476644    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.477020    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.477043    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.477056    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.477069    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30ea652e-3239-40dd-9c46-c826105e70ed-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.477081    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.477705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.477406    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:02:24.479781 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.479718    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "kube-api-access-x2rcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:24.480012 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.479986    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:24.480104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.480067    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy" pod "30ea652e-3239-40dd-9c46-c826105e70ed" (UID: "30ea652e-3239-40dd-9c46-c826105e70ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:24.577974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.577942    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2rcg\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-kube-api-access-x2rcg\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.577974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.577970    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30ea652e-3239-40dd-9c46-c826105e70ed-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.578138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.577981    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30ea652e-3239-40dd-9c46-c826105e70ed-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:24.578138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:24.577989    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30ea652e-3239-40dd-9c46-c826105e70ed-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:25.175884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:25.175802    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"
Nov 01 23:02:25.175884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:25.175843    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4" event={"ID":"30ea652e-3239-40dd-9c46-c826105e70ed","Type":"ContainerDied","Data":"ba2171cec845be78d27bef6f9b7ded5ad88209fb397b55aa005b6ce7aaae73c8"}
Nov 01 23:02:25.176386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:25.175921    4531 scope.go:122] "RemoveContainer" containerID="a5f3a34e0d53fafe5ce608b8db3f5fa503da8c6a610d460fa7110f0139f37c59"
Nov 01 23:02:25.334787 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:25.334761    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"]
Nov 01 23:02:25.383569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:25.383538    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b8fd2-openshift-default-6dbd8766fc-g6ch4"]
Nov 01 23:02:26.817564 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:26.817532    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" path="/var/lib/kubelet/pods/30ea652e-3239-40dd-9c46-c826105e70ed/volumes"
Nov 01 23:02:27.200852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:27.200814    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:31.263181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:31.263141    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:02:31.263784 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:31.263141    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:02:32.395251 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:32.395215    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:36.109862 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.109837    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 23:02:36.120833 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.120815    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 23:02:36.213685 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.213658    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl\" (UniqueName: \"kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl\") pod \"2e2f1c24-981c-4224-834a-1169ee831c91\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") "
Nov 01 23:02:36.213685 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.213687    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume\") pod \"2e2f1c24-981c-4224-834a-1169ee831c91\" (UID: \"2e2f1c24-981c-4224-834a-1169ee831c91\") "
Nov 01 23:02:36.213885 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.213752    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx\" (UniqueName: \"kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx\") pod \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") "
Nov 01 23:02:36.213885 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.213873    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume\") pod \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\" (UID: \"dbfb0e7f-25ba-489f-aac2-258bd6e1966d\") "
Nov 01 23:02:36.214017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.213996    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume" pod "2e2f1c24-981c-4224-834a-1169ee831c91" (UID: "2e2f1c24-981c-4224-834a-1169ee831c91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:36.214267 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.214247    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2f1c24-981c-4224-834a-1169ee831c91-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:36.214339 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.214275    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume" pod "dbfb0e7f-25ba-489f-aac2-258bd6e1966d" (UID: "dbfb0e7f-25ba-489f-aac2-258bd6e1966d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:36.215790 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.215771    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx" pod "dbfb0e7f-25ba-489f-aac2-258bd6e1966d" (UID: "dbfb0e7f-25ba-489f-aac2-258bd6e1966d"). InnerVolumeSpecName "kube-api-access-bwcjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:36.215883 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.215831    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl" pod "2e2f1c24-981c-4224-834a-1169ee831c91" (UID: "2e2f1c24-981c-4224-834a-1169ee831c91"). InnerVolumeSpecName "kube-api-access-c4npl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:36.288337 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.288311    4531 generic.go:356] "Generic (PLEG): container finished" podID="2e2f1c24-981c-4224-834a-1169ee831c91" containerID="de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e" exitCode=0
Nov 01 23:02:36.288470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.288354    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"
Nov 01 23:02:36.288470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.288403    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" event={"ID":"2e2f1c24-981c-4224-834a-1169ee831c91","Type":"ContainerDied","Data":"de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e"}
Nov 01 23:02:36.288470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.288455    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd" event={"ID":"2e2f1c24-981c-4224-834a-1169ee831c91","Type":"ContainerDied","Data":"fe4c075c800bd00af123a6041d39e194abd72df5fa6fdd92b3d434c85ef529c1"}
Nov 01 23:02:36.288600 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.288482    4531 scope.go:122] "RemoveContainer" containerID="de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e"
Nov 01 23:02:36.290097 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.290074    4531 generic.go:356] "Generic (PLEG): container finished" podID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerID="18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9" exitCode=0
Nov 01 23:02:36.290235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.290119    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"
Nov 01 23:02:36.290235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.290168    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" event={"ID":"dbfb0e7f-25ba-489f-aac2-258bd6e1966d","Type":"ContainerDied","Data":"18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9"}
Nov 01 23:02:36.290235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.290217    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t" event={"ID":"dbfb0e7f-25ba-489f-aac2-258bd6e1966d","Type":"ContainerDied","Data":"226f652c009a68e0d0da7b6c6927a52780aaa01c2cc0a7c6edc97bf5334a22b7"}
Nov 01 23:02:36.301120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.301085    4531 scope.go:122] "RemoveContainer" containerID="de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e"
Nov 01 23:02:36.301407 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:36.301384    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e\": container with ID starting with de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e not found: ID does not exist" containerID="de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e"
Nov 01 23:02:36.301462 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.301418    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e"} err="failed to get container status \"de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e\": rpc error: code = NotFound desc = could not find container \"de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e\": container with ID starting with de18e51378232b8ec105d71db48a58cc0b9b2ff64415196c3b9894137c6a952e not found: ID does not exist"
Nov 01 23:02:36.301462 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.301439    4531 scope.go:122] "RemoveContainer" containerID="18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9"
Nov 01 23:02:36.311717 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.311697    4531 scope.go:122] "RemoveContainer" containerID="18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9"
Nov 01 23:02:36.312027 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:36.312007    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9\": container with ID starting with 18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9 not found: ID does not exist" containerID="18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9"
Nov 01 23:02:36.312107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.312033    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9"} err="failed to get container status \"18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9\": rpc error: code = NotFound desc = could not find container \"18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9\": container with ID starting with 18c201626d5d19553ba6a4099bf9aae48aa3e0f1cc94fb84a5a95665fa16dea9 not found: ID does not exist"
Nov 01 23:02:36.315740 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.315712    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:36.315828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.315747    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4npl\" (UniqueName: \"kubernetes.io/projected/2e2f1c24-981c-4224-834a-1169ee831c91-kube-api-access-c4npl\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:36.315828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.315758    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwcjx\" (UniqueName: \"kubernetes.io/projected/dbfb0e7f-25ba-489f-aac2-258bd6e1966d-kube-api-access-bwcjx\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:36.434264 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.434199    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"]
Nov 01 23:02:36.438051 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.438028    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-d6101b-654dbdc749-7zhmd"]
Nov 01 23:02:36.538955 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.538930    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"]
Nov 01 23:02:36.542408 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.542384    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-d6101b-6bf9d9c7cf-j5b4t"]
Nov 01 23:02:36.818806 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.818778    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" path="/var/lib/kubelet/pods/2e2f1c24-981c-4224-834a-1169ee831c91/volumes"
Nov 01 23:02:36.819321 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:36.819304    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" path="/var/lib/kubelet/pods/dbfb0e7f-25ba-489f-aac2-258bd6e1966d/volumes"
Nov 01 23:02:37.965432 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:37.965397    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:38.330868 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:38.330720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:38.476872 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:38.476836    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"]
Nov 01 23:02:38.520754 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:38.520714    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7da9ed-8m5mz"]
Nov 01 23:02:38.535097 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:38.535068    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:38.818207 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:38.818178    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e89e194-2302-4692-94ca-ee9b46406c04" path="/var/lib/kubelet/pods/8e89e194-2302-4692-94ca-ee9b46406c04/volumes"
Nov 01 23:02:38.959787 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:38.959750    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:40.994139 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:40.994109    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:41.206357 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:41.206324    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:41.419855 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:41.419776    4531 scope.go:122] "RemoveContainer" containerID="19605a59df22dba530b321b414a0d4343c1f4703223cb907470f7e3e9fff1c11"
Nov 01 23:02:42.522138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:42.522108    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:42.538471 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:42.538427    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:02:43.489927 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:43.489890    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:45.559533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:45.559501    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:45.759110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:45.759074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:45.855829 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:45.855757    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"]
Nov 01 23:02:45.855988 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:45.855968    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" containerID="cri-o://ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7" gracePeriod=30
Nov 01 23:02:45.953665 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:45.951554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:47.952449 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.952415    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"]
Nov 01 23:02:47.953087 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953068    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main"
Nov 01 23:02:47.953087 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953086    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953100    4531 cpu_manager.go:414] "containerMap: removing container" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953105    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953120    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" containerName="payload-processing"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953125    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" containerName="payload-processing"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953133    4531 cpu_manager.go:414] "containerMap: removing container" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerName="payload-pre-processing"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953139    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerName="payload-pre-processing"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953155    4531 cpu_manager.go:414] "containerMap: removing container" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main"
Nov 01 23:02:47.953220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953159    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main"
Nov 01 23:02:47.953637 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953269    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9d7fb2-7701-49f7-9184-7d85caba1ed1" containerName="main"
Nov 01 23:02:47.953637 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953279    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ea652e-3239-40dd-9c46-c826105e70ed" containerName="istio-proxy"
Nov 01 23:02:47.953637 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953287    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2f1c24-981c-4224-834a-1169ee831c91" containerName="payload-processing"
Nov 01 23:02:47.953637 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953292    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbfb0e7f-25ba-489f-aac2-258bd6e1966d" containerName="payload-pre-processing"
Nov 01 23:02:47.953637 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.953301    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="996218de-84d9-4c3f-a3b5-4a09e7109fe5" containerName="main"
Nov 01 23:02:47.958771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.958748    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:47.979350 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:47.979318    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:48.009696 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.009659    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"]
Nov 01 23:02:48.050313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.050275    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:02:48.050488 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.050371    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:02:48.073802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.072373    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.073802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.072446    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c6nk\" (UniqueName: \"kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.073802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.072584    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.173692 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.173658    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.173905 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.173774    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c6nk\" (UniqueName: \"kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.173969 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.173946    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.174762 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.174716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.176162 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.176141    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.227800 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.227718    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c6nk\" (UniqueName: \"kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.291300 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.278003    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:48.689452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.689426    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"]
Nov 01 23:02:48.692986 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:02:48.692959    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2380590b_0448_4650_bfec_0ee518dfee51.slice/crio-0bcae0d5826dee4b260aa3b7191c2ac6c6d029a3b47828eef79a55a18589513f WatchSource:0}: Error finding container 0bcae0d5826dee4b260aa3b7191c2ac6c6d029a3b47828eef79a55a18589513f: Status 404 returned error can't find the container with id 0bcae0d5826dee4b260aa3b7191c2ac6c6d029a3b47828eef79a55a18589513f
Nov 01 23:02:48.729174 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.729147    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:48.949443 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:48.949411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:49.416430 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:49.416347    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc" event={"ID":"2380590b-0448-4650-bfec-0ee518dfee51","Type":"ContainerStarted","Data":"4251befbc7237ecf994649ffeba62e82aff1a1d68f1422baa5134c83a46c01bd"}
Nov 01 23:02:49.416430 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:49.416404    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc" event={"ID":"2380590b-0448-4650-bfec-0ee518dfee51","Type":"ContainerStarted","Data":"0bcae0d5826dee4b260aa3b7191c2ac6c6d029a3b47828eef79a55a18589513f"}
Nov 01 23:02:49.551560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:49.551511    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc" podStartSLOduration=2.551497674 podStartE2EDuration="2.551497674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:02:49.548807372 +0000 UTC m=+2350.866457371" watchObservedRunningTime="2026-11-01 23:02:49.551497674 +0000 UTC m=+2350.869147674"
Nov 01 23:02:49.657471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:49.657443    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:49.925371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:49.925336    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:50.070568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.070518    4531 patch_prober.go:29] interesting pod/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:02:50.070773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.070599    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:02:50.092879 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.092807    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"]
Nov 01 23:02:50.096661 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.096640    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-584561-7ljbg"]
Nov 01 23:02:50.116275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.116246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:50.436055 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.435974    4531 generic.go:356] "Generic (PLEG): container finished" podID="2380590b-0448-4650-bfec-0ee518dfee51" containerID="4251befbc7237ecf994649ffeba62e82aff1a1d68f1422baa5134c83a46c01bd" exitCode=0
Nov 01 23:02:50.436414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.436060    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc" event={"ID":"2380590b-0448-4650-bfec-0ee518dfee51","Type":"ContainerDied","Data":"4251befbc7237ecf994649ffeba62e82aff1a1d68f1422baa5134c83a46c01bd"}
Nov 01 23:02:50.517362 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.517328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:50.819910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:50.819881    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98295795-d339-4700-a606-3671398b2341" path="/var/lib/kubelet/pods/98295795-d339-4700-a606-3671398b2341/volumes"
Nov 01 23:02:51.146773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.146745    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 23:02:51.229775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.229749    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.229929 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.229797    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.229929 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.229831    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.229929 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.229868    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.229929 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.229926    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.230080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.229998    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.230080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230040    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.230080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230069    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.230241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230081    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:51.230241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230116    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token\") pod \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\" (UID: \"290646e6-1e8b-41ff-9e0b-8d21674fbf9c\") "
Nov 01 23:02:51.230345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230252    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:51.230345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230276    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:51.230549 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230526    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:51.230653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230631    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:51.230719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230667    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.230719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230686    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.230719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230694    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.230719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.230704    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.231989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.231964    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:02:51.232148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.232124    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:02:51.232230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.232213    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "kube-api-access-kqb8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:51.232365 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.232353    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token" pod "290646e6-1e8b-41ff-9e0b-8d21674fbf9c" (UID: "290646e6-1e8b-41ff-9e0b-8d21674fbf9c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:51.332252 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.332223    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.332252 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.332251    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqb8n\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-kube-api-access-kqb8n\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.332454 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.332262    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.332454 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.332271    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.332454 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.332280    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/290646e6-1e8b-41ff-9e0b-8d21674fbf9c-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.445653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.445618    4531 generic.go:356] "Generic (PLEG): container finished" podID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerID="ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7" exitCode=0
Nov 01 23:02:51.446136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.445672    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"
Nov 01 23:02:51.446136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.445706    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" event={"ID":"290646e6-1e8b-41ff-9e0b-8d21674fbf9c","Type":"ContainerDied","Data":"ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7"}
Nov 01 23:02:51.446136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.445794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd" event={"ID":"290646e6-1e8b-41ff-9e0b-8d21674fbf9c","Type":"ContainerDied","Data":"6bd8c8c5508cedeced5181086eb34b53d46062d247648f4cb2e66fc2c2ac41d5"}
Nov 01 23:02:51.446136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.445822    4531 scope.go:122] "RemoveContainer" containerID="ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7"
Nov 01 23:02:51.458323 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.458307    4531 scope.go:122] "RemoveContainer" containerID="4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"
Nov 01 23:02:51.471238 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.471218    4531 scope.go:122] "RemoveContainer" containerID="ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7"
Nov 01 23:02:51.471599 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:51.471563    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7\": container with ID starting with ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7 not found: ID does not exist" containerID="ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7"
Nov 01 23:02:51.471672 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.471613    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7"} err="failed to get container status \"ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7\": rpc error: code = NotFound desc = could not find container \"ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7\": container with ID starting with ce39dcba3bf2e890a5bf72a6353d5c54c349ea4d7eb1441aa96b5aa67f1c07f7 not found: ID does not exist"
Nov 01 23:02:51.471672 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.471640    4531 scope.go:122] "RemoveContainer" containerID="4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"
Nov 01 23:02:51.471966 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:02:51.471946    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb\": container with ID starting with 4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb not found: ID does not exist" containerID="4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"
Nov 01 23:02:51.472034 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.471974    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb"} err="failed to get container status \"4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb\": rpc error: code = NotFound desc = could not find container \"4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb\": container with ID starting with 4165928c39b8e5df8ae88d4b1e3d5f25f22276300303e0fa07d6f484134c05bb not found: ID does not exist"
Nov 01 23:02:51.597532 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.597509    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:51.602338 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.602311    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"]
Nov 01 23:02:51.608981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.608959    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7da9ed-openshift-default-db8d9d575-n65jd"]
Nov 01 23:02:51.635783 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.635756    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk\" (UniqueName: \"kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk\") pod \"2380590b-0448-4650-bfec-0ee518dfee51\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") "
Nov 01 23:02:51.635956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.635939    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca\") pod \"2380590b-0448-4650-bfec-0ee518dfee51\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") "
Nov 01 23:02:51.636046 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.636033    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca\") pod \"2380590b-0448-4650-bfec-0ee518dfee51\" (UID: \"2380590b-0448-4650-bfec-0ee518dfee51\") "
Nov 01 23:02:51.636384 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.636358    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca" pod "2380590b-0448-4650-bfec-0ee518dfee51" (UID: "2380590b-0448-4650-bfec-0ee518dfee51"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:02:51.637715 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.637698    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk" pod "2380590b-0448-4650-bfec-0ee518dfee51" (UID: "2380590b-0448-4650-bfec-0ee518dfee51"). InnerVolumeSpecName "kube-api-access-6c6nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:02:51.637956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.637937    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca" pod "2380590b-0448-4650-bfec-0ee518dfee51" (UID: "2380590b-0448-4650-bfec-0ee518dfee51"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:02:51.737603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.737524    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.737603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.737552    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2380590b-0448-4650-bfec-0ee518dfee51-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:51.737603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:51.737564    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c6nk\" (UniqueName: \"kubernetes.io/projected/2380590b-0448-4650-bfec-0ee518dfee51-kube-api-access-6c6nk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:02:52.455954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:52.455921    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"
Nov 01 23:02:52.456441 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:52.455916    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc" event={"ID":"2380590b-0448-4650-bfec-0ee518dfee51","Type":"ContainerDied","Data":"0bcae0d5826dee4b260aa3b7191c2ac6c6d029a3b47828eef79a55a18589513f"}
Nov 01 23:02:52.456441 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:52.456132    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bcae0d5826dee4b260aa3b7191c2ac6c6d029a3b47828eef79a55a18589513f"
Nov 01 23:02:52.818751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:52.818655    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" path="/var/lib/kubelet/pods/290646e6-1e8b-41ff-9e0b-8d21674fbf9c/volumes"
Nov 01 23:02:55.339421 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:55.339386    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:57.904228 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.904195    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:57.994340 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.994307    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"]
Nov 01 23:02:57.994943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.994925    4531 cpu_manager.go:414] "containerMap: removing container" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy"
Nov 01 23:02:57.994943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.994940    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.994950    4531 cpu_manager.go:414] "containerMap: removing container" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.994956    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.994996    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2380590b-0448-4650-bfec-0ee518dfee51" containerName="revoke-keys"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.995002    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2380590b-0448-4650-bfec-0ee518dfee51" containerName="revoke-keys"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.995114    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.995123    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="290646e6-1e8b-41ff-9e0b-8d21674fbf9c" containerName="istio-proxy"
Nov 01 23:02:57.995149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.995133    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2380590b-0448-4650-bfec-0ee518dfee51" containerName="revoke-keys"
Nov 01 23:02:57.999784 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:57.999761    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.055448 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.055415    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"]
Nov 01 23:02:58.095424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.095398    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:02:58.095602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.095496    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:02:58.133802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.133770    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.133937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.133838    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.134020 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.133996    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9twh\" (UniqueName: \"kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.235296 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.235266    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.235414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.235335    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.235414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.235401    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9twh\" (UniqueName: \"kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.236011 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.235988    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.237489 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.237470    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.295456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.295428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9twh\" (UniqueName: \"kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh\") pod \"maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.316343 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.316318    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:02:58.701471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:58.701438    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"]
Nov 01 23:02:58.706176 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:02:58.706147    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode771518e_795b_4de9_8981_e7b13fe2fd5e.slice/crio-fc56e11de751ddf3daa1e01151e1bc33175e5538d9b471430e055ad579b6e7b8 WatchSource:0}: Error finding container fc56e11de751ddf3daa1e01151e1bc33175e5538d9b471430e055ad579b6e7b8: Status 404 returned error can't find the container with id fc56e11de751ddf3daa1e01151e1bc33175e5538d9b471430e055ad579b6e7b8
Nov 01 23:02:59.054625 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.054593    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"]
Nov 01 23:02:59.158277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.158247    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"]
Nov 01 23:02:59.477960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.477838    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:02:59.579029 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.578988    4531 generic.go:356] "Generic (PLEG): container finished" podID="e771518e-795b-4de9-8981-e7b13fe2fd5e" containerID="6b143d64a2eb51a39aa6159fd2f8ef2bfd19a44c9830beeb2a87009bc4a2446d" exitCode=0
Nov 01 23:02:59.579239 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.579218    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d" event={"ID":"e771518e-795b-4de9-8981-e7b13fe2fd5e","Type":"ContainerDied","Data":"6b143d64a2eb51a39aa6159fd2f8ef2bfd19a44c9830beeb2a87009bc4a2446d"}
Nov 01 23:02:59.579312 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.579273    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d" event={"ID":"e771518e-795b-4de9-8981-e7b13fe2fd5e","Type":"ContainerStarted","Data":"fc56e11de751ddf3daa1e01151e1bc33175e5538d9b471430e055ad579b6e7b8"}
Nov 01 23:02:59.665327 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:02:59.665297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:00.428975 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.428939    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:00.830244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.830221    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:03:00.965758 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.965214    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca\") pod \"e771518e-795b-4de9-8981-e7b13fe2fd5e\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") "
Nov 01 23:03:00.965758 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.965267    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh\" (UniqueName: \"kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh\") pod \"e771518e-795b-4de9-8981-e7b13fe2fd5e\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") "
Nov 01 23:03:00.965758 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.965344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca\") pod \"e771518e-795b-4de9-8981-e7b13fe2fd5e\" (UID: \"e771518e-795b-4de9-8981-e7b13fe2fd5e\") "
Nov 01 23:03:00.966064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.965974    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca" pod "e771518e-795b-4de9-8981-e7b13fe2fd5e" (UID: "e771518e-795b-4de9-8981-e7b13fe2fd5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:00.968024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.967980    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca" pod "e771518e-795b-4de9-8981-e7b13fe2fd5e" (UID: "e771518e-795b-4de9-8981-e7b13fe2fd5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:00.968182 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:00.968106    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh" pod "e771518e-795b-4de9-8981-e7b13fe2fd5e" (UID: "e771518e-795b-4de9-8981-e7b13fe2fd5e"). InnerVolumeSpecName "kube-api-access-k9twh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:01.067317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:01.067254    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:01.067317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:01.067279    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e771518e-795b-4de9-8981-e7b13fe2fd5e-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:01.067317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:01.067288    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9twh\" (UniqueName: \"kubernetes.io/projected/e771518e-795b-4de9-8981-e7b13fe2fd5e-kube-api-access-k9twh\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:01.610383 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:01.610353    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"
Nov 01 23:03:01.610383 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:01.610366    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d" event={"ID":"e771518e-795b-4de9-8981-e7b13fe2fd5e","Type":"ContainerDied","Data":"fc56e11de751ddf3daa1e01151e1bc33175e5538d9b471430e055ad579b6e7b8"}
Nov 01 23:03:01.610992 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:01.610416    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc56e11de751ddf3daa1e01151e1bc33175e5538d9b471430e055ad579b6e7b8"
Nov 01 23:03:02.617657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:02.617617    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:04.055572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.055534    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6" podUID="cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" containerName="payload-processing" containerID="cri-o://3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64" gracePeriod=25
Nov 01 23:03:04.158798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.158759    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x" podUID="a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" containerName="payload-pre-processing" containerID="cri-o://6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db" gracePeriod=25
Nov 01 23:03:04.333469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.332704    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 23:03:04.409407 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.408528    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume\") pod \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") "
Nov 01 23:03:04.409407 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.408662    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l\" (UniqueName: \"kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l\") pod \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\" (UID: \"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078\") "
Nov 01 23:03:04.410102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.410068    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume" pod "cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" (UID: "cf99fe19-3e2c-49bf-a49f-ef95c6e4e078"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:04.411371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.411330    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l" pod "cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" (UID: "cf99fe19-3e2c-49bf-a49f-ef95c6e4e078"). InnerVolumeSpecName "kube-api-access-5z47l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:04.435258 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.435237    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 23:03:04.510346 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.510317    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume\") pod \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") "
Nov 01 23:03:04.510511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.510380    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7\" (UniqueName: \"kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7\") pod \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\" (UID: \"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38\") "
Nov 01 23:03:04.510710 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.510687    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume" pod "a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" (UID: "a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:04.511040 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.511015    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:04.511040 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.511037    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:04.511040 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.511047    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z47l\" (UniqueName: \"kubernetes.io/projected/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078-kube-api-access-5z47l\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:04.512642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.512618    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7" pod "a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" (UID: "a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38"). InnerVolumeSpecName "kube-api-access-js5d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:04.611990 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.611909    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js5d7\" (UniqueName: \"kubernetes.io/projected/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38-kube-api-access-js5d7\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:04.645485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.645456    4531 generic.go:356] "Generic (PLEG): container finished" podID="cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" containerID="3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64" exitCode=0
Nov 01 23:03:04.645657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.645502    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"
Nov 01 23:03:04.645657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.645554    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6" event={"ID":"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078","Type":"ContainerDied","Data":"3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64"}
Nov 01 23:03:04.645657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.645617    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6" event={"ID":"cf99fe19-3e2c-49bf-a49f-ef95c6e4e078","Type":"ContainerDied","Data":"05e0b4a459077b836703c26696e68e41a04625bc1bde8ba4926c8b49b552c95b"}
Nov 01 23:03:04.645657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.645651    4531 scope.go:122] "RemoveContainer" containerID="3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64"
Nov 01 23:03:04.647016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.646996    4531 generic.go:356] "Generic (PLEG): container finished" podID="a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" containerID="6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db" exitCode=0
Nov 01 23:03:04.647133 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.647025    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x" event={"ID":"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38","Type":"ContainerDied","Data":"6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db"}
Nov 01 23:03:04.647133 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.647054    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"
Nov 01 23:03:04.647133 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.647067    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x" event={"ID":"a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38","Type":"ContainerDied","Data":"e308d038eb6d9fbd18a222a3e89f3081c4a1b47a854fa2adfbd8af33b1074bb4"}
Nov 01 23:03:04.656527 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.656376    4531 scope.go:122] "RemoveContainer" containerID="3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64"
Nov 01 23:03:04.656688 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:04.656665    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64\": container with ID starting with 3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64 not found: ID does not exist" containerID="3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64"
Nov 01 23:03:04.656798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.656702    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64"} err="failed to get container status \"3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64\": rpc error: code = NotFound desc = could not find container \"3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64\": container with ID starting with 3e2276fd6b65b70b9d30c8581bcf9063a1f85d6e2b6a86098230d39a318c6e64 not found: ID does not exist"
Nov 01 23:03:04.656798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.656740    4531 scope.go:122] "RemoveContainer" containerID="6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db"
Nov 01 23:03:04.666643 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.666622    4531 scope.go:122] "RemoveContainer" containerID="6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db"
Nov 01 23:03:04.667007 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:04.666985    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db\": container with ID starting with 6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db not found: ID does not exist" containerID="6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db"
Nov 01 23:03:04.667088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.667013    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db"} err="failed to get container status \"6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db\": rpc error: code = NotFound desc = could not find container \"6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db\": container with ID starting with 6977c381df53e025ca9538f62d143a73a12956e56e62632e70c1659e087367db not found: ID does not exist"
Nov 01 23:03:04.802502 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.802463    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"]
Nov 01 23:03:04.806397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.806369    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-a97162-54db5586bf-qxbb6"]
Nov 01 23:03:04.817632 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.817609    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" path="/var/lib/kubelet/pods/cf99fe19-3e2c-49bf-a49f-ef95c6e4e078/volumes"
Nov 01 23:03:04.909657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.909586    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"]
Nov 01 23:03:04.915360 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:04.915335    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-a97162-9b9dcc859-tqd5x"]
Nov 01 23:03:05.624157 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:05.624123    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:06.816560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:06.816532    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" path="/var/lib/kubelet/pods/a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38/volumes"
Nov 01 23:03:09.628270 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:09.628240    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"]
Nov 01 23:03:09.939783 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:09.939679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:10.128658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:10.128621    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:10.808179 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:10.808145    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:12.527184 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:12.527149    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:03:14.628718 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.628677    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerName="payload-pre-processing" containerID="cri-o://096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1" gracePeriod=25
Nov 01 23:03:14.917114 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.917040    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"]
Nov 01 23:03:14.917972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.917954    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" containerName="payload-pre-processing"
Nov 01 23:03:14.918023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.917975    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" containerName="payload-pre-processing"
Nov 01 23:03:14.918023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.917996    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e771518e-795b-4de9-8981-e7b13fe2fd5e" containerName="revoke-keys"
Nov 01 23:03:14.918023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.918003    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e771518e-795b-4de9-8981-e7b13fe2fd5e" containerName="revoke-keys"
Nov 01 23:03:14.918121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.918030    4531 cpu_manager.go:414] "containerMap: removing container" podUID="cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" containerName="payload-processing"
Nov 01 23:03:14.918121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.918038    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" containerName="payload-processing"
Nov 01 23:03:14.918225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.918213    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ba7ec5-d21e-4c2c-97b8-f278fcf4cc38" containerName="payload-pre-processing"
Nov 01 23:03:14.918260 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.918228    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e771518e-795b-4de9-8981-e7b13fe2fd5e" containerName="revoke-keys"
Nov 01 23:03:14.918260 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.918240    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf99fe19-3e2c-49bf-a49f-ef95c6e4e078" containerName="payload-processing"
Nov 01 23:03:14.923775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.923754    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:14.971169 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.971131    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"]
Nov 01 23:03:14.975523 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.975503    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"]
Nov 01 23:03:14.975643 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:14.975590    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.012889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.012861    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:03:15.013008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.012904    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:03:15.035225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.035201    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.035382 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.035251    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2ltm\" (UniqueName: \"kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.035382 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.035370    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.077697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.077656    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"]
Nov 01 23:03:15.131877 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.131848    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"]
Nov 01 23:03:15.136274 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.136252    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.136615 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.136588    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2ltm\" (UniqueName: \"kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.136703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.136648    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj24n\" (UniqueName: \"kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.136783 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.136706    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.136863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.136844    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.136939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.136924    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.137133 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.137114    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.137827 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.137803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.139218 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.139190    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.194028 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.193958    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2ltm\" (UniqueName: \"kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm\") pod \"maas-api-revoke-sub-6c2c73b2-dnltz\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.195081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.195055    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"]
Nov 01 23:03:15.198467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.198445    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"]
Nov 01 23:03:15.203074 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.203057    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.237908 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.237880    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.238034 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.237947    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.238034 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.237995    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.238156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.238084    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88jsw\" (UniqueName: \"kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.238156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.238113    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.238315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.238291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj24n\" (UniqueName: \"kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.238816 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.238788    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.240891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.240872    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.241414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.241402    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:15.252707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.252682    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"]
Nov 01 23:03:15.300776 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.296166    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj24n\" (UniqueName: \"kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n\") pod \"maas-api-revoke-sub-27032c97-xfpct\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.340763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.340010    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.340763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.340052    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.340763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.340100    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.340763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.340145    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88jsw\" (UniqueName: \"kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.340763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.340166    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.340763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.340263    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khvt\" (UniqueName: \"kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.342155 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.342103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.343858 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.343836    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.394452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.394426    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88jsw\" (UniqueName: \"kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw\") pod \"maas-api-revoke-sub-ccdcf4d7-lw75w\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.441705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.441678    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khvt\" (UniqueName: \"kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.441862 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.441833    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.441907 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.441883    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.442568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.442547    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.444130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.444083    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.462177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.462155    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:15.495302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.495279    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khvt\" (UniqueName: \"kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt\") pod \"maas-api-revoke-sub-b616f3e1-4clbc\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.525564 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.525526    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:15.591001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.590718    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:15.652610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.645580    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"]
Nov 01 23:03:15.739650 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.739618    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz" event={"ID":"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1","Type":"ContainerStarted","Data":"6929dbebdb4c881f81622ac2b3966f3b011322d65666186a6468f009ae7e83ad"}
Nov 01 23:03:15.820124 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.820098    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"]
Nov 01 23:03:15.838482 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.838456    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.850165 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.850140    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"]
Nov 01 23:03:15.852315 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:15.852292    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6326600_cdb1_4a78_a51a_dc3dcb76230c.slice/crio-c581c9e81b0d4fe0630d0ddddd3ec9f1e2a0c98b998678ce5f575df7e571c240 WatchSource:0}: Error finding container c581c9e81b0d4fe0630d0ddddd3ec9f1e2a0c98b998678ce5f575df7e571c240: Status 404 returned error can't find the container with id c581c9e81b0d4fe0630d0ddddd3ec9f1e2a0c98b998678ce5f575df7e571c240
Nov 01 23:03:15.857668 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.857638    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.857812 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.857790    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgj7\" (UniqueName: \"kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.857874 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.857844    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.857919 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.857889    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.885909 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.885886    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"]
Nov 01 23:03:15.890787 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.890765    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-55b58e-serving-cert\""
Nov 01 23:03:15.914467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.914443    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"]
Nov 01 23:03:15.915821 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:15.915790    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d9c0bb4_9bf3_473c_9d19_cea9ce4b1350.slice/crio-f97fead2d2822de7ba9937bc1d8ca278002d9d4273b728d5326515c1be7df889 WatchSource:0}: Error finding container f97fead2d2822de7ba9937bc1d8ca278002d9d4273b728d5326515c1be7df889: Status 404 returned error can't find the container with id f97fead2d2822de7ba9937bc1d8ca278002d9d4273b728d5326515c1be7df889
Nov 01 23:03:15.959522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.959492    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgj7\" (UniqueName: \"kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.959633 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.959581    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.959633 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.959606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.959791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.959758    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.960274 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:15.960256    4531 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-serving-cert: secret "maas-api-e2e-api-a-55b58e-serving-cert" not found
Nov 01 23:03:15.960482 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:15.960459    4531 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls podName:8b2d55f6-2ac2-4343-a801-31e2e3b50e57 nodeName:}" failed. No retries permitted until 2026-11-01 23:03:16.460438923 +0000 UTC m=+2377.778088935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls") pod "maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" (UID: "8b2d55f6-2ac2-4343-a801-31e2e3b50e57") : secret "maas-api-e2e-api-a-55b58e-serving-cert" not found
Nov 01 23:03:15.961384 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.961360    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:15.965816 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:15.965760    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:16.009323 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.009290    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"]
Nov 01 23:03:16.062992 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.062957    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgj7\" (UniqueName: \"kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:16.128898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.128865    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:16.323554 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.323524    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:16.466631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.466542    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:16.468802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.468714    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\") pod \"maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:16.750240 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.750207    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct" event={"ID":"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9","Type":"ContainerStarted","Data":"451766247f76e08ee5e417bf4b895de3b445755482e8981805fe2519ad6e0bf2"}
Nov 01 23:03:16.750631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.750270    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct" event={"ID":"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9","Type":"ContainerStarted","Data":"7cc947f29085d41fc55ca34ffc3d92da4966c09320a63a371fd1b6b00f1919b7"}
Nov 01 23:03:16.751953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.751929    4531 generic.go:356] "Generic (PLEG): container finished" podID="a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" containerID="1dcd906697a83448b4c6d10f9edc9f35029ad84a30749b0cdd35d865d9aadf26" exitCode=0
Nov 01 23:03:16.752075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.752002    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz" event={"ID":"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1","Type":"ContainerDied","Data":"1dcd906697a83448b4c6d10f9edc9f35029ad84a30749b0cdd35d865d9aadf26"}
Nov 01 23:03:16.753857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.753833    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc" event={"ID":"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350","Type":"ContainerStarted","Data":"7898789f9312cd7dc4ee30d4ad101c345e1651434431ebfbb2ae50672dfdbd65"}
Nov 01 23:03:16.753990 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.753885    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc" event={"ID":"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350","Type":"ContainerStarted","Data":"f97fead2d2822de7ba9937bc1d8ca278002d9d4273b728d5326515c1be7df889"}
Nov 01 23:03:16.755651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.755629    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w" event={"ID":"f6326600-cdb1-4a78-a51a-dc3dcb76230c","Type":"ContainerStarted","Data":"857c75ed8bf41a23be3b90f0253736970a385c834f6a14f9ccf00beaab2915f1"}
Nov 01 23:03:16.755786 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.755669    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w" event={"ID":"f6326600-cdb1-4a78-a51a-dc3dcb76230c","Type":"ContainerStarted","Data":"c581c9e81b0d4fe0630d0ddddd3ec9f1e2a0c98b998678ce5f575df7e571c240"}
Nov 01 23:03:16.758978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.758953    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:16.862606 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:16.862535    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct" podStartSLOduration=2.862516792 podStartE2EDuration="2.862516792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:16.858825016 +0000 UTC m=+2378.176475017" watchObservedRunningTime="2026-11-01 23:03:16.862516792 +0000 UTC m=+2378.180166794"
Nov 01 23:03:17.076214 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.076167    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w" podStartSLOduration=2.076153558 podStartE2EDuration="2.076153558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:17.072487658 +0000 UTC m=+2378.390137660" watchObservedRunningTime="2026-11-01 23:03:17.076153558 +0000 UTC m=+2378.393803575"
Nov 01 23:03:17.187051 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.186912    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc" podStartSLOduration=2.186892575 podStartE2EDuration="2.186892575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:17.180590819 +0000 UTC m=+2378.498240819" watchObservedRunningTime="2026-11-01 23:03:17.186892575 +0000 UTC m=+2378.504542576"
Nov 01 23:03:17.194201 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.194174    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"]
Nov 01 23:03:17.195471 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:17.195443    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2d55f6_2ac2_4343_a801_31e2e3b50e57.slice/crio-4a73cf522f53b016eb29124be350ef7c93552f5291009775b6bbadee30203b9c WatchSource:0}: Error finding container 4a73cf522f53b016eb29124be350ef7c93552f5291009775b6bbadee30203b9c: Status 404 returned error can't find the container with id 4a73cf522f53b016eb29124be350ef7c93552f5291009775b6bbadee30203b9c
Nov 01 23:03:17.197216 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.197193    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:03:17.765091 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.765059    4531 generic.go:356] "Generic (PLEG): container finished" podID="7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" containerID="7898789f9312cd7dc4ee30d4ad101c345e1651434431ebfbb2ae50672dfdbd65" exitCode=0
Nov 01 23:03:17.765566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.765146    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc" event={"ID":"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350","Type":"ContainerDied","Data":"7898789f9312cd7dc4ee30d4ad101c345e1651434431ebfbb2ae50672dfdbd65"}
Nov 01 23:03:17.766957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.766933    4531 generic.go:356] "Generic (PLEG): container finished" podID="f6326600-cdb1-4a78-a51a-dc3dcb76230c" containerID="857c75ed8bf41a23be3b90f0253736970a385c834f6a14f9ccf00beaab2915f1" exitCode=0
Nov 01 23:03:17.767090 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.766996    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w" event={"ID":"f6326600-cdb1-4a78-a51a-dc3dcb76230c","Type":"ContainerDied","Data":"857c75ed8bf41a23be3b90f0253736970a385c834f6a14f9ccf00beaab2915f1"}
Nov 01 23:03:17.768707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.768682    4531 generic.go:356] "Generic (PLEG): container finished" podID="ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" containerID="451766247f76e08ee5e417bf4b895de3b445755482e8981805fe2519ad6e0bf2" exitCode=0
Nov 01 23:03:17.768828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.768758    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct" event={"ID":"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9","Type":"ContainerDied","Data":"451766247f76e08ee5e417bf4b895de3b445755482e8981805fe2519ad6e0bf2"}
Nov 01 23:03:17.770188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.770166    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" event={"ID":"8b2d55f6-2ac2-4343-a801-31e2e3b50e57","Type":"ContainerStarted","Data":"4a73cf522f53b016eb29124be350ef7c93552f5291009775b6bbadee30203b9c"}
Nov 01 23:03:17.927362 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.927337    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:17.983689 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.983657    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca\") pod \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") "
Nov 01 23:03:17.987452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.984984    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca" pod "a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" (UID: "a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:17.996164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.996135    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca\") pod \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") "
Nov 01 23:03:17.996295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.996247    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm\" (UniqueName: \"kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm\") pod \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\" (UID: \"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1\") "
Nov 01 23:03:17.996774 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.996751    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:17.999910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.999884    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca" pod "a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" (UID: "a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:18.000012 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:17.999945    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm" pod "a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" (UID: "a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1"). InnerVolumeSpecName "kube-api-access-z2ltm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:18.100761 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.100645    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.100761 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.100680    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2ltm\" (UniqueName: \"kubernetes.io/projected/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1-kube-api-access-z2ltm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:18.782216 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.782178    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" event={"ID":"8b2d55f6-2ac2-4343-a801-31e2e3b50e57","Type":"ContainerStarted","Data":"580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492"}
Nov 01 23:03:18.782755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.782428    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:18.783969 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.783938    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"
Nov 01 23:03:18.784095 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.783968    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz" event={"ID":"a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1","Type":"ContainerDied","Data":"6929dbebdb4c881f81622ac2b3966f3b011322d65666186a6468f009ae7e83ad"}
Nov 01 23:03:18.784095 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.784015    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6929dbebdb4c881f81622ac2b3966f3b011322d65666186a6468f009ae7e83ad"
Nov 01 23:03:18.901511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.901456    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" podStartSLOduration=2.932841099 podStartE2EDuration="3.901436654s" totalImagesPullingTime="968.595555ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:18.89119902 +0000 UTC m=+2380.208849022" watchObservedRunningTime="2026-11-01 23:03:18.901436654 +0000 UTC m=+2380.219086657"
Nov 01 23:03:18.992578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:18.992552    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:19.065203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.065172    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:19.070194 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.070163    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:19.147401 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147374    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca\") pod \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") "
Nov 01 23:03:19.147586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147411    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca\") pod \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") "
Nov 01 23:03:19.147586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147447    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n\" (UniqueName: \"kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n\") pod \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") "
Nov 01 23:03:19.147586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147486    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt\" (UniqueName: \"kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt\") pod \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") "
Nov 01 23:03:19.147586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147506    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca\") pod \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") "
Nov 01 23:03:19.147890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147674    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw\" (UniqueName: \"kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw\") pod \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\" (UID: \"f6326600-cdb1-4a78-a51a-dc3dcb76230c\") "
Nov 01 23:03:19.147890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147780    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca\") pod \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\" (UID: \"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9\") "
Nov 01 23:03:19.147890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147787    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca" pod "ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" (UID: "ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:19.147890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147838    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca\") pod \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") "
Nov 01 23:03:19.148132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.147895    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca" pod "f6326600-cdb1-4a78-a51a-dc3dcb76230c" (UID: "f6326600-cdb1-4a78-a51a-dc3dcb76230c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:19.148132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.148018    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca\") pod \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\" (UID: \"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350\") "
Nov 01 23:03:19.148954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.148561    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca" pod "7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" (UID: "7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:19.148954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.148812    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.148954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.148837    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.148954 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.148852    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.150469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.150444    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt" pod "7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" (UID: "7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350"). InnerVolumeSpecName "kube-api-access-7khvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:19.150469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.150457    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca" pod "ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" (UID: "ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:19.150639 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.150478    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n" pod "ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" (UID: "ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9"). InnerVolumeSpecName "kube-api-access-hj24n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:19.150639 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.150485    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw" pod "f6326600-cdb1-4a78-a51a-dc3dcb76230c" (UID: "f6326600-cdb1-4a78-a51a-dc3dcb76230c"). InnerVolumeSpecName "kube-api-access-88jsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:19.150639 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.150553    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca" pod "f6326600-cdb1-4a78-a51a-dc3dcb76230c" (UID: "f6326600-cdb1-4a78-a51a-dc3dcb76230c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:19.151039 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.151018    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca" pod "7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" (UID: "7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:03:19.249583 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.249552    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.249583 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.249578    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.249583 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.249588    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6326600-cdb1-4a78-a51a-dc3dcb76230c-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.249823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.249597    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj24n\" (UniqueName: \"kubernetes.io/projected/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9-kube-api-access-hj24n\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.249823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.249606    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khvt\" (UniqueName: \"kubernetes.io/projected/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350-kube-api-access-7khvt\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.249823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.249616    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88jsw\" (UniqueName: \"kubernetes.io/projected/f6326600-cdb1-4a78-a51a-dc3dcb76230c-kube-api-access-88jsw\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:19.680070 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.680030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:19.794458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.794429    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"
Nov 01 23:03:19.794889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.794453    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc" event={"ID":"7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350","Type":"ContainerDied","Data":"f97fead2d2822de7ba9937bc1d8ca278002d9d4273b728d5326515c1be7df889"}
Nov 01 23:03:19.794889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.794511    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f97fead2d2822de7ba9937bc1d8ca278002d9d4273b728d5326515c1be7df889"
Nov 01 23:03:19.795962 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.795933    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"
Nov 01 23:03:19.796094 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.795981    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w" event={"ID":"f6326600-cdb1-4a78-a51a-dc3dcb76230c","Type":"ContainerDied","Data":"c581c9e81b0d4fe0630d0ddddd3ec9f1e2a0c98b998678ce5f575df7e571c240"}
Nov 01 23:03:19.796094 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.796030    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c581c9e81b0d4fe0630d0ddddd3ec9f1e2a0c98b998678ce5f575df7e571c240"
Nov 01 23:03:19.797644 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.797625    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"
Nov 01 23:03:19.797813 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.797634    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct" event={"ID":"ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9","Type":"ContainerDied","Data":"7cc947f29085d41fc55ca34ffc3d92da4966c09320a63a371fd1b6b00f1919b7"}
Nov 01 23:03:19.797813 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:19.797685    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cc947f29085d41fc55ca34ffc3d92da4966c09320a63a371fd1b6b00f1919b7"
Nov 01 23:03:20.092482 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:20.092421    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:21.343414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:21.343370    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:22.183752 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:22.182704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:22.416240 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:22.416197    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:24.313820 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:24.313776    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:03:24.804583 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:24.804551    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:03:26.247698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:26.247669    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:26.663471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:26.663398    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:26.864332 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:26.864297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:27.256941 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:27.256909    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:27.706271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:27.706238    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:28.368093 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:28.367975    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:28.671010 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:28.670282    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:31.567244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:31.567208    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:32.017802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.017769    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:32.221991 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.221960    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:32.517071 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.517034    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"]
Nov 01 23:03:32.518076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518050    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" containerName="revoke-keys"
Nov 01 23:03:32.518076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518072    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" containerName="revoke-keys"
Nov 01 23:03:32.518280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518121    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" containerName="revoke-keys"
Nov 01 23:03:32.518280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518130    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" containerName="revoke-keys"
Nov 01 23:03:32.518280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518150    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" containerName="revoke-keys"
Nov 01 23:03:32.518280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518158    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" containerName="revoke-keys"
Nov 01 23:03:32.518280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518188    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f6326600-cdb1-4a78-a51a-dc3dcb76230c" containerName="revoke-keys"
Nov 01 23:03:32.518280 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518196    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6326600-cdb1-4a78-a51a-dc3dcb76230c" containerName="revoke-keys"
Nov 01 23:03:32.518467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518316    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" containerName="revoke-keys"
Nov 01 23:03:32.518467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518331    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" containerName="revoke-keys"
Nov 01 23:03:32.518467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518344    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6326600-cdb1-4a78-a51a-dc3dcb76230c" containerName="revoke-keys"
Nov 01 23:03:32.518467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.518355    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" containerName="revoke-keys"
Nov 01 23:03:32.524018 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.523999    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.618390 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.618344    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"]
Nov 01 23:03:32.623121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.623097    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4af44d\""
Nov 01 23:03:32.623400 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.623381    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4af44d-dockercfg-xwj64\""
Nov 01 23:03:32.653201 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.653048    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.653356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.653263    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvnv\" (UniqueName: \"kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv\") pod \"payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.695248 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.695214    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"]
Nov 01 23:03:32.700368 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.700345    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:32.757713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.757306    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.757713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.757411    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvnv\" (UniqueName: \"kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv\") pod \"payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.763587 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.758862    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.763762 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.763643    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"]
Nov 01 23:03:32.813777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.813687    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvnv\" (UniqueName: \"kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv\") pod \"payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.849902 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.849872    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:32.858395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.858368    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgd9c\" (UniqueName: \"kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c\") pod \"payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:32.858506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.858424    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:32.960238 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.960208    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgd9c\" (UniqueName: \"kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c\") pod \"payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:32.960399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.960287    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:32.961019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:32.960994    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:33.014229 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.014204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgd9c\" (UniqueName: \"kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c\") pod \"payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:33.014390 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.014375    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:33.230328 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.230258    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"]
Nov 01 23:03:33.234073 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:33.234045    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0d40d6_f0a9_43d3_9d01_7c5c36708ef3.slice/crio-1efb4c47c2891c3b4b9de4e21cd977b5140e327aaa4a9b21db97a55dfd383465 WatchSource:0}: Error finding container 1efb4c47c2891c3b4b9de4e21cd977b5140e327aaa4a9b21db97a55dfd383465: Status 404 returned error can't find the container with id 1efb4c47c2891c3b4b9de4e21cd977b5140e327aaa4a9b21db97a55dfd383465
Nov 01 23:03:33.387900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.387837    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"]
Nov 01 23:03:33.391147 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:33.391122    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd454cd9d_fea6_400b_b53e_f6ecc01d0619.slice/crio-672fb9839a88d3d9505f3a816ccde9f177f46245985961c1935a5aeba766c420 WatchSource:0}: Error finding container 672fb9839a88d3d9505f3a816ccde9f177f46245985961c1935a5aeba766c420: Status 404 returned error can't find the container with id 672fb9839a88d3d9505f3a816ccde9f177f46245985961c1935a5aeba766c420
Nov 01 23:03:33.506840 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.506809    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:33.697025 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.696992    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:33.940289 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.940249    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" event={"ID":"d454cd9d-fea6-400b-b53e-f6ecc01d0619","Type":"ContainerStarted","Data":"1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98"}
Nov 01 23:03:33.940481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.940323    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:33.940481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.940350    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" event={"ID":"d454cd9d-fea6-400b-b53e-f6ecc01d0619","Type":"ContainerStarted","Data":"672fb9839a88d3d9505f3a816ccde9f177f46245985961c1935a5aeba766c420"}
Nov 01 23:03:33.941904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.941877    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr" event={"ID":"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3","Type":"ContainerStarted","Data":"90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61"}
Nov 01 23:03:33.942024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.941921    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr" event={"ID":"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3","Type":"ContainerStarted","Data":"1efb4c47c2891c3b4b9de4e21cd977b5140e327aaa4a9b21db97a55dfd383465"}
Nov 01 23:03:33.942024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:33.941949    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:34.056149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.056062    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" podStartSLOduration=2.056046721 podStartE2EDuration="2.056046721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:34.051311338 +0000 UTC m=+2395.368961338" watchObservedRunningTime="2026-11-01 23:03:34.056046721 +0000 UTC m=+2395.373696723"
Nov 01 23:03:34.160487 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.160436    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr" podStartSLOduration=2.160421668 podStartE2EDuration="2.160421668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:34.158312736 +0000 UTC m=+2395.475962737" watchObservedRunningTime="2026-11-01 23:03:34.160421668 +0000 UTC m=+2395.478071670"
Nov 01 23:03:34.312712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.312625    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:03:34.909761 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.909700    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 23:03:34.950713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.950684    4531 generic.go:356] "Generic (PLEG): container finished" podID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerID="096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1" exitCode=0
Nov 01 23:03:34.950899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.950758    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"
Nov 01 23:03:34.950899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.950783    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" event={"ID":"9e351981-372f-4a68-ad4a-5cb2fe8ae610","Type":"ContainerDied","Data":"096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1"}
Nov 01 23:03:34.950899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.950844    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh" event={"ID":"9e351981-372f-4a68-ad4a-5cb2fe8ae610","Type":"ContainerDied","Data":"6535231b7db4a88c0a9dcce5f9c43c2e23c145a87f0b187a4583ab6cc1fd057e"}
Nov 01 23:03:34.950899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.950876    4531 scope.go:122] "RemoveContainer" containerID="096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1"
Nov 01 23:03:34.961859 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.961840    4531 scope.go:122] "RemoveContainer" containerID="096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1"
Nov 01 23:03:34.962159 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:34.962136    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1\": container with ID starting with 096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1 not found: ID does not exist" containerID="096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1"
Nov 01 23:03:34.962248 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:34.962165    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1"} err="failed to get container status \"096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1\": rpc error: code = NotFound desc = could not find container \"096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1\": container with ID starting with 096e8fad9ecb7bdab558c868f3d796c02a1102ecd4622f623e1cca3d33e3afa1 not found: ID does not exist"
Nov 01 23:03:35.086478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.086448    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm\" (UniqueName: \"kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm\") pod \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") "
Nov 01 23:03:35.086657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.086496    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume\") pod \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\" (UID: \"9e351981-372f-4a68-ad4a-5cb2fe8ae610\") "
Nov 01 23:03:35.086979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.086940    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume" pod "9e351981-372f-4a68-ad4a-5cb2fe8ae610" (UID: "9e351981-372f-4a68-ad4a-5cb2fe8ae610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:03:35.087170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.087148    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e351981-372f-4a68-ad4a-5cb2fe8ae610-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:35.088983 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.088958    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm" pod "9e351981-372f-4a68-ad4a-5cb2fe8ae610" (UID: "9e351981-372f-4a68-ad4a-5cb2fe8ae610"). InnerVolumeSpecName "kube-api-access-5g8gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:03:35.188195 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.188143    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g8gm\" (UniqueName: \"kubernetes.io/projected/9e351981-372f-4a68-ad4a-5cb2fe8ae610-kube-api-access-5g8gm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:03:35.366841 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.366818    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"]
Nov 01 23:03:35.372857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:35.372834    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-e71d55-d4fd45d99-ldmnh"]
Nov 01 23:03:36.818713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:36.818680    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" path="/var/lib/kubelet/pods/9e351981-372f-4a68-ad4a-5cb2fe8ae610/volumes"
Nov 01 23:03:36.864396 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:36.864357    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:37.053365 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.053328    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:37.244996 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.244905    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:37.420597 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.420568    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"]
Nov 01 23:03:37.421245 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.421229    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerName="payload-pre-processing"
Nov 01 23:03:37.421302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.421247    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerName="payload-pre-processing"
Nov 01 23:03:37.421371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.421362    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e351981-372f-4a68-ad4a-5cb2fe8ae610" containerName="payload-pre-processing"
Nov 01 23:03:37.425927 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.425910    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.483640 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.483613    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"]
Nov 01 23:03:37.541362 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.541277    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cf46c7\""
Nov 01 23:03:37.612911 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.552808    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbf4z\" (UniqueName: \"kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z\") pod \"payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.613171 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.613143    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.613828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.592678    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"]
Nov 01 23:03:37.618536 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.609024    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cf46c7-dockercfg-wfgjq\""
Nov 01 23:03:37.631121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.631096    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.640054 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.640012    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"]
Nov 01 23:03:37.726170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.726123    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.726339 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.726280    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbf4z\" (UniqueName: \"kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z\") pod \"payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.726392 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.726353    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.726450 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.726392    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdfv\" (UniqueName: \"kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv\") pod \"payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.727154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.727130    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.779957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.779932    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbf4z\" (UniqueName: \"kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z\") pod \"payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:37.827719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.827658    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.828070 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.827754    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdfv\" (UniqueName: \"kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv\") pod \"payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.828339 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.828318    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.882203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.882177    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdfv\" (UniqueName: \"kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv\") pod \"payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:37.949036 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:37.949011    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:38.040356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.040127    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:38.330685 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.330652    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"]
Nov 01 23:03:38.334281 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:38.334257    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52362a8f_e57d_41dc_805c_bcfd024f143d.slice/crio-e6ce563f05641047335f352608a23a220f775a02bcda3864426f6c41d976e167 WatchSource:0}: Error finding container e6ce563f05641047335f352608a23a220f775a02bcda3864426f6c41d976e167: Status 404 returned error can't find the container with id e6ce563f05641047335f352608a23a220f775a02bcda3864426f6c41d976e167
Nov 01 23:03:38.430810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.430787    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"]
Nov 01 23:03:38.433922 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:03:38.433899    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b7bb7df_9c18_4e39_aa88_606a4da1a139.slice/crio-e2782ad048d835833bdc5446be6fb0b8922010a4e2d43360a457e0c668bf7cc4 WatchSource:0}: Error finding container e2782ad048d835833bdc5446be6fb0b8922010a4e2d43360a457e0c668bf7cc4: Status 404 returned error can't find the container with id e2782ad048d835833bdc5446be6fb0b8922010a4e2d43360a457e0c668bf7cc4
Nov 01 23:03:38.992456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.992414    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" event={"ID":"52362a8f-e57d-41dc-805c-bcfd024f143d","Type":"ContainerStarted","Data":"c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea"}
Nov 01 23:03:38.992962 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.992480    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:38.992962 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.992502    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" event={"ID":"52362a8f-e57d-41dc-805c-bcfd024f143d","Type":"ContainerStarted","Data":"e6ce563f05641047335f352608a23a220f775a02bcda3864426f6c41d976e167"}
Nov 01 23:03:38.994472 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.994435    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" event={"ID":"0b7bb7df-9c18-4e39-aa88-606a4da1a139","Type":"ContainerStarted","Data":"4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2"}
Nov 01 23:03:38.994619 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.994496    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:38.994619 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:38.994515    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" event={"ID":"0b7bb7df-9c18-4e39-aa88-606a4da1a139","Type":"ContainerStarted","Data":"e2782ad048d835833bdc5446be6fb0b8922010a4e2d43360a457e0c668bf7cc4"}
Nov 01 23:03:39.119649 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:39.119593    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" podStartSLOduration=2.119580409 podStartE2EDuration="2.119580409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:39.117515304 +0000 UTC m=+2400.435165305" watchObservedRunningTime="2026-11-01 23:03:39.119580409 +0000 UTC m=+2400.437230431"
Nov 01 23:03:39.261397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:39.261267    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" podStartSLOduration=2.261246625 podStartE2EDuration="2.261246625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:03:39.25297973 +0000 UTC m=+2400.570629736" watchObservedRunningTime="2026-11-01 23:03:39.261246625 +0000 UTC m=+2400.578896627"
Nov 01 23:03:39.585791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:39.585682    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:39.795413 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:39.795377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:41.113040 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:41.113010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:41.313933 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:41.313900    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:41.940549 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:41.940419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:42.528880 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:03:42.528840    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:03:44.954112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:44.954079    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:03:44.954548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:44.954162    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:03:46.780754 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:46.780697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:46.985956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:46.985922    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:50.006358 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:50.006326    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:03:50.006839 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:50.006406    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:03:50.953501 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:50.953461    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:51.276081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:51.276003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:51.488845 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:51.488807    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:52.597881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:52.597840    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:52.787041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:52.787010    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:52.977459 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:52.977430    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:53.038857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:53.038827    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"]
Nov 01 23:03:53.087311 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:53.087283    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-a97162-jv5vc"]
Nov 01 23:03:53.481573 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:53.481539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:54.522972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:54.522933    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:54.817941 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:54.817866    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2380590b-0448-4650-bfec-0ee518dfee51" path="/var/lib/kubelet/pods/2380590b-0448-4650-bfec-0ee518dfee51/volumes"
Nov 01 23:03:57.721987 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:57.721937    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:59.404492 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:59.404454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:59.616209 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:59.616173    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:03:59.826110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:03:59.823079    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:00.021335 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.021299    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:00.744349 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.744317    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"]
Nov 01 23:04:00.751080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.751054    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:00.803553 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.803527    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"]
Nov 01 23:04:00.843996 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.843969    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:04:00.844135 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.844102    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:04:00.901746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.901136    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:00.901746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.901283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:00.901746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:00.901330    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btz7q\" (UniqueName: \"kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.003168 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.003089    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.003168 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.003134    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btz7q\" (UniqueName: \"kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.003357 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.003300    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.003889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.003868    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.005800 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.005760    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.056720 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.056695    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btz7q\" (UniqueName: \"kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.065519 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.065497    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:01.444129 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:01.444050    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"]
Nov 01 23:04:01.447410 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:04:01.447383    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbb8dce_b3cb_4a23_93d6_c42416bc070f.slice/crio-581f9d37996fbf703b5f516184e6609ca2da281cf3ef8dbfd236a5ae54861247 WatchSource:0}: Error finding container 581f9d37996fbf703b5f516184e6609ca2da281cf3ef8dbfd236a5ae54861247: Status 404 returned error can't find the container with id 581f9d37996fbf703b5f516184e6609ca2da281cf3ef8dbfd236a5ae54861247
Nov 01 23:04:02.197148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:02.197106    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg" event={"ID":"4cbb8dce-b3cb-4a23-93d6-c42416bc070f","Type":"ContainerStarted","Data":"a44d4748839a9351ecc28ad99c2d4784122da861c15c9e2ebe91094d589d2e74"}
Nov 01 23:04:02.197578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:02.197181    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg" event={"ID":"4cbb8dce-b3cb-4a23-93d6-c42416bc070f","Type":"ContainerStarted","Data":"581f9d37996fbf703b5f516184e6609ca2da281cf3ef8dbfd236a5ae54861247"}
Nov 01 23:04:02.307588 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:02.307523    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg" podStartSLOduration=2.307502884 podStartE2EDuration="2.307502884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:02.305107581 +0000 UTC m=+2423.622757582" watchObservedRunningTime="2026-11-01 23:04:02.307502884 +0000 UTC m=+2423.625152886"
Nov 01 23:04:02.710678 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:02.710643    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:03.209138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:03.209105    4531 generic.go:356] "Generic (PLEG): container finished" podID="4cbb8dce-b3cb-4a23-93d6-c42416bc070f" containerID="a44d4748839a9351ecc28ad99c2d4784122da861c15c9e2ebe91094d589d2e74" exitCode=0
Nov 01 23:04:03.209645 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:03.209155    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg" event={"ID":"4cbb8dce-b3cb-4a23-93d6-c42416bc070f","Type":"ContainerDied","Data":"a44d4748839a9351ecc28ad99c2d4784122da861c15c9e2ebe91094d589d2e74"}
Nov 01 23:04:04.374042 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.374011    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:04.549205 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.549130    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca\") pod \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") "
Nov 01 23:04:04.549345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.549262    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q\" (UniqueName: \"kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q\") pod \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") "
Nov 01 23:04:04.549345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.549319    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca\") pod \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\" (UID: \"4cbb8dce-b3cb-4a23-93d6-c42416bc070f\") "
Nov 01 23:04:04.549544 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.549515    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca" pod "4cbb8dce-b3cb-4a23-93d6-c42416bc070f" (UID: "4cbb8dce-b3cb-4a23-93d6-c42416bc070f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:04.549887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.549863    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:04.551356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.551332    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca" pod "4cbb8dce-b3cb-4a23-93d6-c42416bc070f" (UID: "4cbb8dce-b3cb-4a23-93d6-c42416bc070f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:04.551487 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.551468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q" pod "4cbb8dce-b3cb-4a23-93d6-c42416bc070f" (UID: "4cbb8dce-b3cb-4a23-93d6-c42416bc070f"). InnerVolumeSpecName "kube-api-access-btz7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:04.650680 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.650644    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btz7q\" (UniqueName: \"kubernetes.io/projected/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-kube-api-access-btz7q\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:04.650680 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:04.650676    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cbb8dce-b3cb-4a23-93d6-c42416bc070f-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:05.226183 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:05.226154    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"
Nov 01 23:04:05.226542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:05.226147    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg" event={"ID":"4cbb8dce-b3cb-4a23-93d6-c42416bc070f","Type":"ContainerDied","Data":"581f9d37996fbf703b5f516184e6609ca2da281cf3ef8dbfd236a5ae54861247"}
Nov 01 23:04:05.226542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:05.226279    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581f9d37996fbf703b5f516184e6609ca2da281cf3ef8dbfd236a5ae54861247"
Nov 01 23:04:08.066946 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:08.066908    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:12.470774 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:12.470706    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:12.541099 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:04:12.540972    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:04:12.721636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:12.721544    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:12.913703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:12.913660    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:14.322044 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:14.322003    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:16.853643 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:16.853610    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:17.153082 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:17.152903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:17.299613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:17.299562    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"]
Nov 01 23:04:17.340888 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:17.340854    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-2b0866-dxbtg"]
Nov 01 23:04:17.762841 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:17.762807    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:18.819658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:18.819619    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbb8dce-b3cb-4a23-93d6-c42416bc070f" path="/var/lib/kubelet/pods/4cbb8dce-b3cb-4a23-93d6-c42416bc070f/volumes"
Nov 01 23:04:18.822534 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:18.822508    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:20.064226 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:20.064194    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:20.569480 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:20.569440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:20.781530 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:20.781494    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:20.982946 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:20.982906    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"]
Nov 01 23:04:21.027253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:21.027220    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-e71d55-nbs7d"]
Nov 01 23:04:21.417099 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:21.417021    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:21.713382 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:21.713354    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:21.930471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:21.930434    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:22.138637 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:22.138553    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:22.168542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:22.168511    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"]
Nov 01 23:04:22.217142 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:22.217111    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-d6101b-6k55c"]
Nov 01 23:04:22.590717 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:22.590683    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:22.817965 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:22.817929    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2" path="/var/lib/kubelet/pods/1139fab9-c1ee-4e3e-aa72-a07e4d49cdf2/volumes"
Nov 01 23:04:22.818404 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:22.818390    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e771518e-795b-4de9-8981-e7b13fe2fd5e" path="/var/lib/kubelet/pods/e771518e-795b-4de9-8981-e7b13fe2fd5e/volumes"
Nov 01 23:04:24.456081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:24.456050    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:24.690765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:24.690705    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:24.908613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:24.908532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:25.398037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.398009    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:25.685064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.684993    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"]
Nov 01 23:04:25.686025 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.686005    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4cbb8dce-b3cb-4a23-93d6-c42416bc070f" containerName="revoke-keys"
Nov 01 23:04:25.686077 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.686028    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbb8dce-b3cb-4a23-93d6-c42416bc070f" containerName="revoke-keys"
Nov 01 23:04:25.686192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.686180    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbb8dce-b3cb-4a23-93d6-c42416bc070f" containerName="revoke-keys"
Nov 01 23:04:25.692070 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.692051    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.746924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.746900    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"]
Nov 01 23:04:25.754207 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.754184    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.754342 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.754233    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk4ns\" (UniqueName: \"kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.754342 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.754304    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.830382 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.830337    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:04:25.830543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.830410    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:04:25.855164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.855142    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.855279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.855191    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk4ns\" (UniqueName: \"kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.855279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.855226    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.855830 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.855808    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.857612 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.857590    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:25.909335 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:25.909307    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk4ns\" (UniqueName: \"kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns\") pod \"maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:26.006406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:26.006369    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:26.394220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:26.394194    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"]
Nov 01 23:04:26.397457 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:04:26.397427    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4394327_557a_4598_9e5a_4b7c722da562.slice/crio-6c8254202e14e949cb38c88f8e4d2abb8f82c278d0025ea6c60f3b1200432f67 WatchSource:0}: Error finding container 6c8254202e14e949cb38c88f8e4d2abb8f82c278d0025ea6c60f3b1200432f67: Status 404 returned error can't find the container with id 6c8254202e14e949cb38c88f8e4d2abb8f82c278d0025ea6c60f3b1200432f67
Nov 01 23:04:27.415609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:27.415572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:27.420196 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:27.420170    4531 generic.go:356] "Generic (PLEG): container finished" podID="b4394327-557a-4598-9e5a-4b7c722da562" containerID="c920667b32916fbaaec9e30fee928801fb418e8cd610786f2d44c4b83b0c158e" exitCode=0
Nov 01 23:04:27.420365 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:27.420251    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z" event={"ID":"b4394327-557a-4598-9e5a-4b7c722da562","Type":"ContainerDied","Data":"c920667b32916fbaaec9e30fee928801fb418e8cd610786f2d44c4b83b0c158e"}
Nov 01 23:04:27.420365 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:27.420327    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z" event={"ID":"b4394327-557a-4598-9e5a-4b7c722da562","Type":"ContainerStarted","Data":"6c8254202e14e949cb38c88f8e4d2abb8f82c278d0025ea6c60f3b1200432f67"}
Nov 01 23:04:27.616151 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:27.616071    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:27.814753 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:27.814674    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:28.595660 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.595635    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:28.693808 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.693703    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns\" (UniqueName: \"kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns\") pod \"b4394327-557a-4598-9e5a-4b7c722da562\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") "
Nov 01 23:04:28.693989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.693865    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca\") pod \"b4394327-557a-4598-9e5a-4b7c722da562\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") "
Nov 01 23:04:28.693989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.693930    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca\") pod \"b4394327-557a-4598-9e5a-4b7c722da562\" (UID: \"b4394327-557a-4598-9e5a-4b7c722da562\") "
Nov 01 23:04:28.694270 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.694243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca" pod "b4394327-557a-4598-9e5a-4b7c722da562" (UID: "b4394327-557a-4598-9e5a-4b7c722da562"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:28.694465 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.694440    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:28.695918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.695896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns" pod "b4394327-557a-4598-9e5a-4b7c722da562" (UID: "b4394327-557a-4598-9e5a-4b7c722da562"). InnerVolumeSpecName "kube-api-access-vk4ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:28.696079 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.696059    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca" pod "b4394327-557a-4598-9e5a-4b7c722da562" (UID: "b4394327-557a-4598-9e5a-4b7c722da562"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:28.795584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.795549    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk4ns\" (UniqueName: \"kubernetes.io/projected/b4394327-557a-4598-9e5a-4b7c722da562-kube-api-access-vk4ns\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:28.795771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:28.795601    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b4394327-557a-4598-9e5a-4b7c722da562-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:29.192056 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.192015    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:29.420278 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.420232    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:29.445834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.445806    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z" event={"ID":"b4394327-557a-4598-9e5a-4b7c722da562","Type":"ContainerDied","Data":"6c8254202e14e949cb38c88f8e4d2abb8f82c278d0025ea6c60f3b1200432f67"}
Nov 01 23:04:29.445999 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.445897    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"
Nov 01 23:04:29.447044 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.446616    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8254202e14e949cb38c88f8e4d2abb8f82c278d0025ea6c60f3b1200432f67"
Nov 01 23:04:29.486287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.486246    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"]
Nov 01 23:04:29.486466 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.486429    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" containerID="cri-o://3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994" gracePeriod=30
Nov 01 23:04:29.640757 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.640706    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:29.844850 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.844776    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:29.892174 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.892148    4531 patch_prober.go:29] interesting pod/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:04:29.892320 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:29.892195    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:04:30.117784 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:30.117690    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"]
Nov 01 23:04:30.130942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:30.130912    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"]
Nov 01 23:04:30.332000 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:30.331966    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:30.520346 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:30.520316    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:31.172403 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:31.172353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:31.431394 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:31.431311    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:32.573682 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:32.573641    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:32.765472 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:32.765432    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:34.797923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.797897    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 23:04:34.873917 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.873888    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.873917 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.873925    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874175 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.873972    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874175 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874061    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874401 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874342    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.874401 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874386    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:34.874671 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874642    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874711    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874789    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874880    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.874995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.874932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs\") pod \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\" (UID: \"c6907aea-9a35-4bfc-9cef-d29a0a56c5db\") "
Nov 01 23:04:34.875384 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.875357    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.876136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876110    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.876465 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876443    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.876555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876472    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.876555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876486    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.876555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876499    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.876762 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876644    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.877046 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.876986    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:34.877144 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.877120    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:04:34.878219 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.878192    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "kube-api-access-hzhtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:34.879112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.879074    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token" pod "c6907aea-9a35-4bfc-9cef-d29a0a56c5db" (UID: "c6907aea-9a35-4bfc-9cef-d29a0a56c5db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:34.903960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.903921    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:34.978149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.978064    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzhtf\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-kube-api-access-hzhtf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.978149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.978099    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.978149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.978110    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.978149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.978120    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:34.978149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:34.978128    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6907aea-9a35-4bfc-9cef-d29a0a56c5db-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:35.112177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.112110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:35.324191 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.324161    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:35.515928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.515893    4531 generic.go:356] "Generic (PLEG): container finished" podID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerID="3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994" exitCode=0
Nov 01 23:04:35.516134 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.515958    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"
Nov 01 23:04:35.516134 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.515988    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" event={"ID":"c6907aea-9a35-4bfc-9cef-d29a0a56c5db","Type":"ContainerDied","Data":"3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994"}
Nov 01 23:04:35.516134 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.516046    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq" event={"ID":"c6907aea-9a35-4bfc-9cef-d29a0a56c5db","Type":"ContainerDied","Data":"ef85c1ff2a528782a227b9fe463b44978b1e3b10086e11d290de0288de85e45d"}
Nov 01 23:04:35.516134 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.516079    4531 scope.go:122] "RemoveContainer" containerID="3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994"
Nov 01 23:04:35.528451 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.528431    4531 scope.go:122] "RemoveContainer" containerID="3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994"
Nov 01 23:04:35.528690 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:04:35.528672    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994\": container with ID starting with 3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994 not found: ID does not exist" containerID="3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994"
Nov 01 23:04:35.528746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.528698    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994"} err="failed to get container status \"3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994\": rpc error: code = NotFound desc = could not find container \"3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994\": container with ID starting with 3496a51f6c0f3bbaf04c17de535e85929d78843623dc2a0bb232133a895fa994 not found: ID does not exist"
Nov 01 23:04:35.674558 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.674527    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"]
Nov 01 23:04:35.721359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:35.721325    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-d6101b-openshift-default-646b5bb74c-m4dnq"]
Nov 01 23:04:36.818406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:36.818377    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" path="/var/lib/kubelet/pods/c6907aea-9a35-4bfc-9cef-d29a0a56c5db/volumes"
Nov 01 23:04:37.909747 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:37.909697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:38.517088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:38.517052    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:38.703818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:38.703780    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:39.906422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:39.906382    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:40.086276 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:40.086243    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:41.648674 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:41.648647    4531 scope.go:122] "RemoveContainer" containerID="8e76e9735f11afffe58bef59e7a8c05a3d54279aa65b91863f844d43c627e040"
Nov 01 23:04:41.658887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:41.658864    4531 scope.go:122] "RemoveContainer" containerID="3c93aed735564957213b41a5c62a90ee372d42e84b45dfeec5c6dbaac8a30f84"
Nov 01 23:04:41.660830 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:41.660792    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:41.670069 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:41.670047    4531 scope.go:122] "RemoveContainer" containerID="45020eed86ce448f0f4363508c2026f159c390a832e619eefdf6cde8b04d9627"
Nov 01 23:04:42.526780 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:04:42.526719    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:04:42.868975 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:42.868895    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:43.050555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:43.050518    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:43.229108 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:43.229073    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:43.457952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:43.457905    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:43.644960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:43.644879    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:45.141900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.141848    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main" containerID="cri-o://fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c" gracePeriod=45
Nov 01 23:04:45.155178 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.155140    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main" containerID="cri-o://7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef" gracePeriod=45
Nov 01 23:04:45.480244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.480216    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 23:04:45.483509 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.483489    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 23:04:45.609352 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609319    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache\") pod \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") "
Nov 01 23:04:45.609525 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609417    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir\") pod \"364690f7-d8dc-4592-91cd-26fa27bf3aad\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") "
Nov 01 23:04:45.609525 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609508    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs\" (UniqueName: \"kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs\") pod \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") "
Nov 01 23:04:45.609663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609611    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home\") pod \"364690f7-d8dc-4592-91cd-26fa27bf3aad\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") "
Nov 01 23:04:45.609663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609648    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache\") pod \"364690f7-d8dc-4592-91cd-26fa27bf3aad\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") "
Nov 01 23:04:45.609792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609668    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache" pod "c5e89d4d-7fb0-4c36-873b-580ba2fa0468" (UID: "c5e89d4d-7fb0-4c36-873b-580ba2fa0468"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.609792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609709    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs\") pod \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") "
Nov 01 23:04:45.609792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609763    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir\") pod \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") "
Nov 01 23:04:45.609961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609790    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir" pod "364690f7-d8dc-4592-91cd-26fa27bf3aad" (UID: "364690f7-d8dc-4592-91cd-26fa27bf3aad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.609961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609809    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs\" (UniqueName: \"kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs\") pod \"364690f7-d8dc-4592-91cd-26fa27bf3aad\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") "
Nov 01 23:04:45.609961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609882    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm\") pod \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") "
Nov 01 23:04:45.609961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609921    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9\" (UniqueName: \"kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9\") pod \"364690f7-d8dc-4592-91cd-26fa27bf3aad\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") "
Nov 01 23:04:45.609961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609947    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm\") pod \"364690f7-d8dc-4592-91cd-26fa27bf3aad\" (UID: \"364690f7-d8dc-4592-91cd-26fa27bf3aad\") "
Nov 01 23:04:45.610204 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.609978    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home\") pod \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\" (UID: \"c5e89d4d-7fb0-4c36-873b-580ba2fa0468\") "
Nov 01 23:04:45.610204 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.610030    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home" pod "364690f7-d8dc-4592-91cd-26fa27bf3aad" (UID: "364690f7-d8dc-4592-91cd-26fa27bf3aad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.610481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.610463    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.610481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.610484    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.610642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.610498    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.610818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.610709    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home" pod "c5e89d4d-7fb0-4c36-873b-580ba2fa0468" (UID: "c5e89d4d-7fb0-4c36-873b-580ba2fa0468"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.614777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.613366    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs" pod "364690f7-d8dc-4592-91cd-26fa27bf3aad" (UID: "364690f7-d8dc-4592-91cd-26fa27bf3aad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:45.614777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.613369    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache" pod "364690f7-d8dc-4592-91cd-26fa27bf3aad" (UID: "364690f7-d8dc-4592-91cd-26fa27bf3aad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.614777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.613371    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir" pod "c5e89d4d-7fb0-4c36-873b-580ba2fa0468" (UID: "c5e89d4d-7fb0-4c36-873b-580ba2fa0468"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.614777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.613435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm" pod "c5e89d4d-7fb0-4c36-873b-580ba2fa0468" (UID: "c5e89d4d-7fb0-4c36-873b-580ba2fa0468"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.614777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.613552    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs" pod "c5e89d4d-7fb0-4c36-873b-580ba2fa0468" (UID: "c5e89d4d-7fb0-4c36-873b-580ba2fa0468"). InnerVolumeSpecName "kube-api-access-5wgrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619372    4531 generic.go:356] "Generic (PLEG): container finished" podID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerID="7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef" exitCode=0
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619387    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs" pod "c5e89d4d-7fb0-4c36-873b-580ba2fa0468" (UID: "c5e89d4d-7fb0-4c36-873b-580ba2fa0468"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619503    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619559    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" event={"ID":"364690f7-d8dc-4592-91cd-26fa27bf3aad","Type":"ContainerDied","Data":"7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef"}
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619596    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b" event={"ID":"364690f7-d8dc-4592-91cd-26fa27bf3aad","Type":"ContainerDied","Data":"3f876dc930a90b83af7c2342cdc26f0c0289213b206ec474c0753af53e3a55b2"}
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619618    4531 scope.go:122] "RemoveContainer" containerID="7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef"
Nov 01 23:04:45.619768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.619697    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm" pod "364690f7-d8dc-4592-91cd-26fa27bf3aad" (UID: "364690f7-d8dc-4592-91cd-26fa27bf3aad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:04:45.620287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.620261    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9" pod "364690f7-d8dc-4592-91cd-26fa27bf3aad" (UID: "364690f7-d8dc-4592-91cd-26fa27bf3aad"). InnerVolumeSpecName "kube-api-access-bq5b9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:45.622555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.622533    4531 generic.go:356] "Generic (PLEG): container finished" podID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerID="fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c" exitCode=0
Nov 01 23:04:45.622667 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.622612    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" event={"ID":"c5e89d4d-7fb0-4c36-873b-580ba2fa0468","Type":"ContainerDied","Data":"fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c"}
Nov 01 23:04:45.622667 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.622633    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw" event={"ID":"c5e89d4d-7fb0-4c36-873b-580ba2fa0468","Type":"ContainerDied","Data":"4e8f1830821c0484a3c7f5ebed60a386fd1f22b5311ad8f273d9e930b9b6340a"}
Nov 01 23:04:45.622920 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.622676    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"
Nov 01 23:04:45.639240 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.639220    4531 scope.go:122] "RemoveContainer" containerID="7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef"
Nov 01 23:04:45.639591 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:04:45.639540    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef\": container with ID starting with 7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef not found: ID does not exist" containerID="7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef"
Nov 01 23:04:45.639682 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.639602    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef"} err="failed to get container status \"7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef\": rpc error: code = NotFound desc = could not find container \"7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef\": container with ID starting with 7b68b1f8d11ee26f378585909398878da519d56d1094840c2e696d30dc08a6ef not found: ID does not exist"
Nov 01 23:04:45.639682 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.639627    4531 scope.go:122] "RemoveContainer" containerID="fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c"
Nov 01 23:04:45.650169 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.650147    4531 scope.go:122] "RemoveContainer" containerID="fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c"
Nov 01 23:04:45.650499 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:04:45.650474    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c\": container with ID starting with fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c not found: ID does not exist" containerID="fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c"
Nov 01 23:04:45.650585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.650508    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c"} err="failed to get container status \"fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c\": rpc error: code = NotFound desc = could not find container \"fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c\": container with ID starting with fbec8d4d801f9facd44c99ea1dbe4f6287e3ad9a1b395427aa963d432ae7056c not found: ID does not exist"
Nov 01 23:04:45.711467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711405    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wgrs\" (UniqueName: \"kubernetes.io/projected/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-kube-api-access-5wgrs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711670    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711707    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711738    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711753    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/364690f7-d8dc-4592-91cd-26fa27bf3aad-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711765    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711780    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq5b9\" (UniqueName: \"kubernetes.io/projected/364690f7-d8dc-4592-91cd-26fa27bf3aad-kube-api-access-bq5b9\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711794    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/364690f7-d8dc-4592-91cd-26fa27bf3aad-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.711881 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.711806    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e89d4d-7fb0-4c36-873b-580ba2fa0468-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:45.816152 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.815609    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"]
Nov 01 23:04:45.822228 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:45.822199    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/distinct-2-w0-c54de0-kserve-8fc654759-nxvvw"]
Nov 01 23:04:46.062484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:46.062404    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"]
Nov 01 23:04:46.069338 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:46.069310    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c54de0/premium-w0-c54de0-kserve-5669f65b57-9px6b"]
Nov 01 23:04:46.817535 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:46.817504    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" path="/var/lib/kubelet/pods/364690f7-d8dc-4592-91cd-26fa27bf3aad/volumes"
Nov 01 23:04:46.817991 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:46.817975    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" path="/var/lib/kubelet/pods/c5e89d4d-7fb0-4c36-873b-580ba2fa0468/volumes"
Nov 01 23:04:48.028371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:48.028338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:48.464866 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:48.464834    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:52.114422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.114345    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"]
Nov 01 23:04:52.115030 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115012    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy"
Nov 01 23:04:52.115080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115031    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy"
Nov 01 23:04:52.115080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115051    4531 cpu_manager.go:414] "containerMap: removing container" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main"
Nov 01 23:04:52.115080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115057    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main"
Nov 01 23:04:52.115080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115066    4531 cpu_manager.go:414] "containerMap: removing container" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main"
Nov 01 23:04:52.115080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115070    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main"
Nov 01 23:04:52.115243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115096    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b4394327-557a-4598-9e5a-4b7c722da562" containerName="revoke-keys"
Nov 01 23:04:52.115243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115101    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4394327-557a-4598-9e5a-4b7c722da562" containerName="revoke-keys"
Nov 01 23:04:52.115243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115189    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4394327-557a-4598-9e5a-4b7c722da562" containerName="revoke-keys"
Nov 01 23:04:52.115243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115201    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6907aea-9a35-4bfc-9cef-d29a0a56c5db" containerName="istio-proxy"
Nov 01 23:04:52.115243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115209    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e89d4d-7fb0-4c36-873b-580ba2fa0468" containerName="main"
Nov 01 23:04:52.115243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.115215    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="364690f7-d8dc-4592-91cd-26fa27bf3aad" containerName="main"
Nov 01 23:04:52.120569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.120551    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.191496 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.191465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tk5f\" (UniqueName: \"kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.191797 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.191756    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.191927 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.191806    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.217223 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.217200    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:04:52.217510 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.217489    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:04:52.224617 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.224598    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"]
Nov 01 23:04:52.229010 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.228993    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.231364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.231323    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"]
Nov 01 23:04:52.276809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.276781    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"]
Nov 01 23:04:52.292396 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.292362    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.292548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.292400    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tk5f\" (UniqueName: \"kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.292548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.292427    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnc9x\" (UniqueName: \"kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.292675 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.292596    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.292675 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.292622    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.292829 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.292682    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.293217 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.293195    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.295495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.295453    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.346526 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.346499    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tk5f\" (UniqueName: \"kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f\") pod \"maas-api-revoke-sub-e0a4fbfd-xqfhv\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.393816 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.393737    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.393816 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.393803    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.394014 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.393861    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnc9x\" (UniqueName: \"kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.394597 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.394565    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.396220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.396191    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.434509 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.434484    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:52.447470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.447446    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnc9x\" (UniqueName: \"kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x\") pod \"maas-api-revoke-sub-15353b44-r8lx9\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.545935 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.545890    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:52.821297 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.821269    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"]
Nov 01 23:04:52.822853 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:04:52.822827    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6340ce_216c_4d41_a580_dc613c28e826.slice/crio-35f022af2c4ca310034d46fc587818fd7565c1c0e3d1f1035b9af3a2b678e0af WatchSource:0}: Error finding container 35f022af2c4ca310034d46fc587818fd7565c1c0e3d1f1035b9af3a2b678e0af: Status 404 returned error can't find the container with id 35f022af2c4ca310034d46fc587818fd7565c1c0e3d1f1035b9af3a2b678e0af
Nov 01 23:04:52.932766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:52.932717    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"]
Nov 01 23:04:52.936529 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:04:52.936502    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6a9a915_755c_49ef_a807_0ecf172ab418.slice/crio-9eeb85a26f8e00422b0a8bdfbedc106339889eb1b287c734288b60bb8041fa04 WatchSource:0}: Error finding container 9eeb85a26f8e00422b0a8bdfbedc106339889eb1b287c734288b60bb8041fa04: Status 404 returned error can't find the container with id 9eeb85a26f8e00422b0a8bdfbedc106339889eb1b287c734288b60bb8041fa04
Nov 01 23:04:53.038541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.038479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:53.717346 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.717293    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9" event={"ID":"e6a9a915-755c-49ef-a807-0ecf172ab418","Type":"ContainerStarted","Data":"604fbe3ab83dc6c7c79036ab061e37e064e2a7a2f04caf327cb0295c129f3d62"}
Nov 01 23:04:53.717839 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.717367    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9" event={"ID":"e6a9a915-755c-49ef-a807-0ecf172ab418","Type":"ContainerStarted","Data":"9eeb85a26f8e00422b0a8bdfbedc106339889eb1b287c734288b60bb8041fa04"}
Nov 01 23:04:53.719150 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.719124    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv" event={"ID":"2e6340ce-216c-4d41-a580-dc613c28e826","Type":"ContainerStarted","Data":"0e800ba8a5ad337541161bfba2eab5b9208c257b55aae45b1c14d3c8a5c86cfc"}
Nov 01 23:04:53.719288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.719172    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv" event={"ID":"2e6340ce-216c-4d41-a580-dc613c28e826","Type":"ContainerStarted","Data":"35f022af2c4ca310034d46fc587818fd7565c1c0e3d1f1035b9af3a2b678e0af"}
Nov 01 23:04:53.949287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.949241    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv" podStartSLOduration=1.949228074 podStartE2EDuration="1.949228074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:53.947675776 +0000 UTC m=+2475.265325779" watchObservedRunningTime="2026-11-01 23:04:53.949228074 +0000 UTC m=+2475.266878074"
Nov 01 23:04:53.951715 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:53.951671    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9" podStartSLOduration=1.951656281 podStartE2EDuration="1.951656281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:04:53.84190995 +0000 UTC m=+2475.159559951" watchObservedRunningTime="2026-11-01 23:04:53.951656281 +0000 UTC m=+2475.269306284"
Nov 01 23:04:54.729109 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:54.729074    4531 generic.go:356] "Generic (PLEG): container finished" podID="e6a9a915-755c-49ef-a807-0ecf172ab418" containerID="604fbe3ab83dc6c7c79036ab061e37e064e2a7a2f04caf327cb0295c129f3d62" exitCode=0
Nov 01 23:04:54.729557 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:54.729127    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9" event={"ID":"e6a9a915-755c-49ef-a807-0ecf172ab418","Type":"ContainerDied","Data":"604fbe3ab83dc6c7c79036ab061e37e064e2a7a2f04caf327cb0295c129f3d62"}
Nov 01 23:04:54.730883 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:54.730860    4531 generic.go:356] "Generic (PLEG): container finished" podID="2e6340ce-216c-4d41-a580-dc613c28e826" containerID="0e800ba8a5ad337541161bfba2eab5b9208c257b55aae45b1c14d3c8a5c86cfc" exitCode=0
Nov 01 23:04:54.731016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:54.730982    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv" event={"ID":"2e6340ce-216c-4d41-a580-dc613c28e826","Type":"ContainerDied","Data":"0e800ba8a5ad337541161bfba2eab5b9208c257b55aae45b1c14d3c8a5c86cfc"}
Nov 01 23:04:55.911130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:55.911105    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:55.939474 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:55.939452    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:56.036075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036000    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca\") pod \"e6a9a915-755c-49ef-a807-0ecf172ab418\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") "
Nov 01 23:04:56.036399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036370    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca\") pod \"e6a9a915-755c-49ef-a807-0ecf172ab418\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") "
Nov 01 23:04:56.036584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036433    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x\" (UniqueName: \"kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x\") pod \"e6a9a915-755c-49ef-a807-0ecf172ab418\" (UID: \"e6a9a915-755c-49ef-a807-0ecf172ab418\") "
Nov 01 23:04:56.036584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca\") pod \"2e6340ce-216c-4d41-a580-dc613c28e826\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") "
Nov 01 23:04:56.036584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036535    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca\") pod \"2e6340ce-216c-4d41-a580-dc613c28e826\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") "
Nov 01 23:04:56.036584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036564    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f\" (UniqueName: \"kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f\") pod \"2e6340ce-216c-4d41-a580-dc613c28e826\" (UID: \"2e6340ce-216c-4d41-a580-dc613c28e826\") "
Nov 01 23:04:56.036889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.036683    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca" pod "e6a9a915-755c-49ef-a807-0ecf172ab418" (UID: "e6a9a915-755c-49ef-a807-0ecf172ab418"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:56.037147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.037059    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca" pod "2e6340ce-216c-4d41-a580-dc613c28e826" (UID: "2e6340ce-216c-4d41-a580-dc613c28e826"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:04:56.037147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.037123    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:56.039097 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.039073    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f" pod "2e6340ce-216c-4d41-a580-dc613c28e826" (UID: "2e6340ce-216c-4d41-a580-dc613c28e826"). InnerVolumeSpecName "kube-api-access-5tk5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:56.039224 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.039100    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca" pod "e6a9a915-755c-49ef-a807-0ecf172ab418" (UID: "e6a9a915-755c-49ef-a807-0ecf172ab418"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:56.039224 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.039176    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x" pod "e6a9a915-755c-49ef-a807-0ecf172ab418" (UID: "e6a9a915-755c-49ef-a807-0ecf172ab418"). InnerVolumeSpecName "kube-api-access-jnc9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:04:56.039413 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.039383    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca" pod "2e6340ce-216c-4d41-a580-dc613c28e826" (UID: "2e6340ce-216c-4d41-a580-dc613c28e826"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:04:56.138956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.138928    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6a9a915-755c-49ef-a807-0ecf172ab418-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:56.138956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.138954    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnc9x\" (UniqueName: \"kubernetes.io/projected/e6a9a915-755c-49ef-a807-0ecf172ab418-kube-api-access-jnc9x\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:56.139201 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.138974    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:56.139201 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.138983    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e6340ce-216c-4d41-a580-dc613c28e826-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:56.139201 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.138994    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tk5f\" (UniqueName: \"kubernetes.io/projected/2e6340ce-216c-4d41-a580-dc613c28e826-kube-api-access-5tk5f\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:04:56.748953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.748920    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"
Nov 01 23:04:56.748953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.748942    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9" event={"ID":"e6a9a915-755c-49ef-a807-0ecf172ab418","Type":"ContainerDied","Data":"9eeb85a26f8e00422b0a8bdfbedc106339889eb1b287c734288b60bb8041fa04"}
Nov 01 23:04:56.749178 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.749001    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9eeb85a26f8e00422b0a8bdfbedc106339889eb1b287c734288b60bb8041fa04"
Nov 01 23:04:56.750485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.750469    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"
Nov 01 23:04:56.750485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.750475    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv" event={"ID":"2e6340ce-216c-4d41-a580-dc613c28e826","Type":"ContainerDied","Data":"35f022af2c4ca310034d46fc587818fd7565c1c0e3d1f1035b9af3a2b678e0af"}
Nov 01 23:04:56.750624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:56.750514    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f022af2c4ca310034d46fc587818fd7565c1c0e3d1f1035b9af3a2b678e0af"
Nov 01 23:04:57.541919 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:57.541883    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:57.732424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:57.732390    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:04:57.916175 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:04:57.916095    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:02.615512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:02.615416    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:06.870651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:06.870609    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:07.197285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:07.197242    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:07.488360 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:07.488276    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:08.223640 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:08.223606    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:12.345544 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:12.345497    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:12.526380 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:05:12.526347    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:05:14.222312 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:14.222277    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:14.447436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:14.447403    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:17.234993 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:17.234902    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:21.998912 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:21.998877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:26.886175 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:26.886113    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:31.669382 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:31.669323    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:36.411574 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:36.411516    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:41.257378 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:41.257344    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:41.456591 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:41.456556    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:41.724543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:41.724505    4531 scope.go:122] "RemoveContainer" containerID="917d55bd33860e9d931ddb7ed14d0cc1ebc717551bfd6a1f368c917c85ceb93e"
Nov 01 23:05:42.004297 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:42.004197    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:42.526592 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:05:42.526559    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:05:44.317599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:44.317542    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:46.554508 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:46.554469    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:48.090538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:48.090509    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"]
Nov 01 23:05:48.094692 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:48.094664    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"]
Nov 01 23:05:48.099921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:48.099898    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-359228b8-p7gwp"]
Nov 01 23:05:48.103978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:48.103955    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d21a208-n9pxf"]
Nov 01 23:05:48.818284 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:48.818250    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a4f430-c80c-4dc6-aadf-f75e7ee47d5b" path="/var/lib/kubelet/pods/32a4f430-c80c-4dc6-aadf-f75e7ee47d5b/volumes"
Nov 01 23:05:48.818797 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:48.818768    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc73e051-4f5e-4416-bc8b-65183c0eb84e" path="/var/lib/kubelet/pods/cc73e051-4f5e-4416-bc8b-65183c0eb84e/volumes"
Nov 01 23:05:51.180944 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.180896    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:51.273066 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.273027    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"]
Nov 01 23:05:51.273309 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.273285    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy" containerID="cri-o://e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48" gracePeriod=30
Nov 01 23:05:51.629974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.629946    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"]
Nov 01 23:05:51.630595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.630581    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2e6340ce-216c-4d41-a580-dc613c28e826" containerName="revoke-keys"
Nov 01 23:05:51.630649 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.630596    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6340ce-216c-4d41-a580-dc613c28e826" containerName="revoke-keys"
Nov 01 23:05:51.630649 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.630623    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e6a9a915-755c-49ef-a807-0ecf172ab418" containerName="revoke-keys"
Nov 01 23:05:51.630649 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.630628    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a9a915-755c-49ef-a807-0ecf172ab418" containerName="revoke-keys"
Nov 01 23:05:51.630783 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.630703    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a9a915-755c-49ef-a807-0ecf172ab418" containerName="revoke-keys"
Nov 01 23:05:51.630783 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.630715    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6340ce-216c-4d41-a580-dc613c28e826" containerName="revoke-keys"
Nov 01 23:05:51.634583 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.634568    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.687353 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.687325    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-675d1d\"/\"openshift-service-ca.crt\""
Nov 01 23:05:51.688315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.687672    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-675d1d\"/\"kube-root-ca.crt\""
Nov 01 23:05:51.688418 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.687807    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-675d1d\"/\"default-dockercfg-wrw9k\""
Nov 01 23:05:51.688627 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.688599    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-675d1d\"/\"test-model-675d1d-kserve-self-signed-certs\""
Nov 01 23:05:51.692577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.692556    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"]
Nov 01 23:05:51.721164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.721141    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.721272 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.721183    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.721442 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.721418    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.721493 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.721476    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxvpn\" (UniqueName: \"kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.721603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.721585    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.721687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.721625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.792653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.792607    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:51.822379 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822344    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.822589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822394    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxvpn\" (UniqueName: \"kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.822589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822468    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.822589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822493    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.822589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822535    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.822589 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822577    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.822902 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.822846    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.824293 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.823209    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.824293 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.823340    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.825613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.825594    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.825754 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.825716    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.877020 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.876989    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxvpn\" (UniqueName: \"kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn\") pod \"test-model-675d1d-kserve-5867f55b49-klts6\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") " pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:51.948080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:51.948045    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:52.337001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:52.336930    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"]
Nov 01 23:05:52.340835 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:05:52.340806    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bc54405_3858_4828_8b79_8c0b35bc40f8.slice/crio-120e92ae6acb09453ea221ad5a3f898d50b2d23ab57b661f13d06b3550de15a9 WatchSource:0}: Error finding container 120e92ae6acb09453ea221ad5a3f898d50b2d23ab57b661f13d06b3550de15a9: Status 404 returned error can't find the container with id 120e92ae6acb09453ea221ad5a3f898d50b2d23ab57b661f13d06b3550de15a9
Nov 01 23:05:53.241898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:53.241849    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6" event={"ID":"1bc54405-3858-4828-8b79-8c0b35bc40f8","Type":"ContainerStarted","Data":"25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c"}
Nov 01 23:05:53.242096 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:53.241913    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6" event={"ID":"1bc54405-3858-4828-8b79-8c0b35bc40f8","Type":"ContainerStarted","Data":"120e92ae6acb09453ea221ad5a3f898d50b2d23ab57b661f13d06b3550de15a9"}
Nov 01 23:05:53.352258 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:53.352187    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6" podStartSLOduration=2.352167881 podStartE2EDuration="2.352167881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:05:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:05:53.349883683 +0000 UTC m=+2534.667533684" watchObservedRunningTime="2026-11-01 23:05:53.352167881 +0000 UTC m=+2534.669817881"
Nov 01 23:05:53.948269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:53.948236    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:05:56.023767 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.023679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:05:56.537628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.537604    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 23:05:56.694876 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.694845    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.694896    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.694981    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695018    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695065    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695318 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695100    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695318 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695182    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695318 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695251    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695318 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695262    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.695805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695350    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:05:56.695805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695342    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.695805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695332    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy\") pod \"81aa20da-82c2-42f0-91c1-a547c05f615e\" (UID: \"81aa20da-82c2-42f0-91c1-a547c05f615e\") "
Nov 01 23:05:56.695805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695437    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.695805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.695657    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.696269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.696067    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81aa20da-82c2-42f0-91c1-a547c05f615e-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.696269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.696083    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.696269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.696093    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.696269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.696103    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.696269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.696116    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.698036 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.698002    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:05:56.698161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.698117    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "kube-api-access-6rpxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:56.698207 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.698150    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:05:56.698295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.698274    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token" pod "81aa20da-82c2-42f0-91c1-a547c05f615e" (UID: "81aa20da-82c2-42f0-91c1-a547c05f615e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:05:56.797102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.797069    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rpxp\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-kube-api-access-6rpxp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.797102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.797097    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.797102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.797107    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:56.797316 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:56.797116    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81aa20da-82c2-42f0-91c1-a547c05f615e-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:05:57.302939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.302908    4531 generic.go:356] "Generic (PLEG): container finished" podID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerID="e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48" exitCode=0
Nov 01 23:05:57.303399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.302993    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"
Nov 01 23:05:57.303399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.302998    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" event={"ID":"81aa20da-82c2-42f0-91c1-a547c05f615e","Type":"ContainerDied","Data":"e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48"}
Nov 01 23:05:57.303399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.303065    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n" event={"ID":"81aa20da-82c2-42f0-91c1-a547c05f615e","Type":"ContainerDied","Data":"76dd1b221963bd9769f5e09c845b0ee9816ccde8b78c901a1d0f0394e892c376"}
Nov 01 23:05:57.303399 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.303093    4531 scope.go:122] "RemoveContainer" containerID="e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48"
Nov 01 23:05:57.314595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.314578    4531 scope.go:122] "RemoveContainer" containerID="e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48"
Nov 01 23:05:57.314901 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:05:57.314884    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48\": container with ID starting with e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48 not found: ID does not exist" containerID="e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48"
Nov 01 23:05:57.314953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.314911    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48"} err="failed to get container status \"e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48\": rpc error: code = NotFound desc = could not find container \"e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48\": container with ID starting with e990889a26d76c3e3b8d14b8ded00b6b43b50e5bf2166185717a245e946dbe48 not found: ID does not exist"
Nov 01 23:05:57.461603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.461569    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"]
Nov 01 23:05:57.509623 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:57.509595    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-a6dd2f-openshift-default-5fdcb97887-swx8n"]
Nov 01 23:05:58.817572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:05:58.817543    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" path="/var/lib/kubelet/pods/81aa20da-82c2-42f0-91c1-a547c05f615e/volumes"
Nov 01 23:06:00.848797 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:00.848763    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:01.213027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:01.212994    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:01.949172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:01.949139    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:06:01.962615 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:01.962587    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:06:02.363764 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:02.363677    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:06:05.329669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.329632    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"]
Nov 01 23:06:05.330120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.329857    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" podUID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" containerName="maas-api" containerID="cri-o://56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70" gracePeriod=30
Nov 01 23:06:05.493168 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.493137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:05.621052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.621029    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 23:06:05.707702 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.707563    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\") pod \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") "
Nov 01 23:06:05.707936 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.707713    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf\" (UniqueName: \"kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf\") pod \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") "
Nov 01 23:06:05.708015 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.707932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca\") pod \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") "
Nov 01 23:06:05.708015 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.707994    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs\") pod \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\" (UID: \"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5\") "
Nov 01 23:06:05.709519 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.709468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca" pod "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" (UID: "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:06:05.713775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.711653    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs" pod "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" (UID: "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:06:05.714392 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.714367    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf" pod "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" (UID: "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5"). InnerVolumeSpecName "kube-api-access-84hwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:06:05.716140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.716104    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls" pod "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" (UID: "ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:06:05.809368 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.809335    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-openshift-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:05.809368 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.809366    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:05.809577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.809376    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-maas-api-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:05.809577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:05.809388    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84hwf\" (UniqueName: \"kubernetes.io/projected/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5-kube-api-access-84hwf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:06.390287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.390248    4531 generic.go:356] "Generic (PLEG): container finished" podID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" containerID="56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70" exitCode=0
Nov 01 23:06:06.390287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.390296    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"
Nov 01 23:06:06.390865 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.390332    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" event={"ID":"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5","Type":"ContainerDied","Data":"56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70"}
Nov 01 23:06:06.390865 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.390396    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" event={"ID":"ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5","Type":"ContainerDied","Data":"13682ddd613b7952170f0cf8f05441e514ecea2f6ab624e0bd5e890aff507a20"}
Nov 01 23:06:06.390865 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.390430    4531 scope.go:122] "RemoveContainer" containerID="56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70"
Nov 01 23:06:06.404398 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.404376    4531 scope.go:122] "RemoveContainer" containerID="56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70"
Nov 01 23:06:06.404673 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:06:06.404648    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70\": container with ID starting with 56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70 not found: ID does not exist" containerID="56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70"
Nov 01 23:06:06.404782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.404684    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70"} err="failed to get container status \"56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70\": rpc error: code = NotFound desc = could not find container \"56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70\": container with ID starting with 56aacee9e90ed5f03f973b0d0d90445100d8ffd843966381d5025a4079f3ff70 not found: ID does not exist"
Nov 01 23:06:06.571344 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.571314    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"]
Nov 01 23:06:06.582851 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.582821    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r"]
Nov 01 23:06:06.817019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:06.816987    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" path="/var/lib/kubelet/pods/ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5/volumes"
Nov 01 23:06:07.027571 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:07.027538    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:07.237962 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:07.237932    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:07.437110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:07.437078    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:07.603431 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:07.603356    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"]
Nov 01 23:06:07.651713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:07.651673    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c54de0-wmp6z"]
Nov 01 23:06:08.052869 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:08.052842    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:08.524591 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:08.524545    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c54de0-79546cf568-rkd9r" podUID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.73:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:06:08.816923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:08.816849    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4394327-557a-4598-9e5a-4b7c722da562" path="/var/lib/kubelet/pods/b4394327-557a-4598-9e5a-4b7c722da562/volumes"
Nov 01 23:06:12.431056 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:12.431022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:12.526182 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:06:12.526146    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:06:14.308937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:14.308889    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:14.536058 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:14.536024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:14.573412 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:14.573330    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"]
Nov 01 23:06:14.574396 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:14.573577    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy" containerID="cri-o://2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d" gracePeriod=30
Nov 01 23:06:14.787458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:14.787419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:17.255047 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:17.255007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:19.846053 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.846030    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 23:06:19.969868 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.969838    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.969907    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.969966    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970009    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970038    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970068    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970107    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970226    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970228    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.970277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970255    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket\") pod \"86dec67a-933b-46da-b3dd-5cfc53f02093\" (UID: \"86dec67a-933b-46da-b3dd-5cfc53f02093\") "
Nov 01 23:06:19.970534 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970501    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:06:19.970856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970646    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.970856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970782    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.970856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970804    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.970856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970809    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.970856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970817    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86dec67a-933b-46da-b3dd-5cfc53f02093-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:19.971254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.970870    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.972958 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.972864    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "kube-api-access-qq5xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:06:19.972958 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.972929    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:06:19.973473 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.973446    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:06:19.974072 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:19.974050    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token" pod "86dec67a-933b-46da-b3dd-5cfc53f02093" (UID: "86dec67a-933b-46da-b3dd-5cfc53f02093"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:06:20.072503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.072467    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.072503 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.072501    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.072697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.072514    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.072697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.072526    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.072697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.072538    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86dec67a-933b-46da-b3dd-5cfc53f02093-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.072697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.072550    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5xk\" (UniqueName: \"kubernetes.io/projected/86dec67a-933b-46da-b3dd-5cfc53f02093-kube-api-access-qq5xk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:06:20.509429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.509400    4531 generic.go:356] "Generic (PLEG): container finished" podID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerID="2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d" exitCode=0
Nov 01 23:06:20.509609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.509452    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"
Nov 01 23:06:20.509609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.509492    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" event={"ID":"86dec67a-933b-46da-b3dd-5cfc53f02093","Type":"ContainerDied","Data":"2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d"}
Nov 01 23:06:20.509609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.509560    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz" event={"ID":"86dec67a-933b-46da-b3dd-5cfc53f02093","Type":"ContainerDied","Data":"5a484660d9cd9a9e022d11375b0ed7d9eb66b4c4bdc6b284c235bbf6f615560b"}
Nov 01 23:06:20.509609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.509592    4531 scope.go:122] "RemoveContainer" containerID="2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d"
Nov 01 23:06:20.521247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.521232    4531 scope.go:122] "RemoveContainer" containerID="2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d"
Nov 01 23:06:20.521516 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:06:20.521498    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d\": container with ID starting with 2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d not found: ID does not exist" containerID="2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d"
Nov 01 23:06:20.521569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.521522    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d"} err="failed to get container status \"2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d\": rpc error: code = NotFound desc = could not find container \"2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d\": container with ID starting with 2e670101b13d05389c5cf93601c8c5dd2a61e4223ae2e0f78a585a302ec1e58d not found: ID does not exist"
Nov 01 23:06:20.668447 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.668418    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"]
Nov 01 23:06:20.674316 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.674292    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c54de0-openshift-default-6f696d8c55-r2rqz"]
Nov 01 23:06:20.818673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:20.818603    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" path="/var/lib/kubelet/pods/86dec67a-933b-46da-b3dd-5cfc53f02093/volumes"
Nov 01 23:06:22.021313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:22.021278    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:26.881743 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:26.881704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:30.033704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:30.033663    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:31.637332 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:31.637296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:36.466546 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:36.466507    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:36.790778 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:36.790658    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:40.018292 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018262    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"]
Nov 01 23:06:40.018876 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018861    4531 cpu_manager.go:414] "containerMap: removing container" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy"
Nov 01 23:06:40.018942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018878    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy"
Nov 01 23:06:40.018942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018901    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" containerName="maas-api"
Nov 01 23:06:40.018942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018906    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" containerName="maas-api"
Nov 01 23:06:40.018942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018923    4531 cpu_manager.go:414] "containerMap: removing container" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy"
Nov 01 23:06:40.018942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.018929    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy"
Nov 01 23:06:40.019126 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.019015    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="86dec67a-933b-46da-b3dd-5cfc53f02093" containerName="istio-proxy"
Nov 01 23:06:40.019126 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.019030    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="81aa20da-82c2-42f0-91c1-a547c05f615e" containerName="istio-proxy"
Nov 01 23:06:40.019126 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.019039    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd4cda4-c874-4922-a2fc-a39b4cfc2fe5" containerName="maas-api"
Nov 01 23:06:40.022792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.022772    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.082917 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.082885    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"]
Nov 01 23:06:40.099989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.099961    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.099989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.099992    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvpgr\" (UniqueName: \"kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.100230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.100035    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.100230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.100166    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.100230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.100200    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.100426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.100273    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.164996 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.164933    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf46c7\"/\"test-model-cf46c7-kserve-self-signed-certs\""
Nov 01 23:06:40.166520 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.166491    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-cf46c7\"/\"default-dockercfg-r7p7l\""
Nov 01 23:06:40.166683 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.166522    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf46c7\"/\"openshift-service-ca.crt\""
Nov 01 23:06:40.166683 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.166634    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-cf46c7\"/\"kube-root-ca.crt\""
Nov 01 23:06:40.172587 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.172564    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:40.201853 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.201826    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.201933    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.201962    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvpgr\" (UniqueName: \"kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202150 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.202107    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202218 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.202195    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.202246    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202336 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.202293    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202401 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.202359    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.202584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.202563    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.204755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.204711    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.205480 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.205458    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.256449 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.256416    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvpgr\" (UniqueName: \"kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr\") pod \"test-model-cf46c7-kserve-6778b5fb5c-sf7tg\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") " pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.335448 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.335362    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:40.735896 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:40.735859    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"]
Nov 01 23:06:40.740253 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:06:40.740215    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod49f56cfd_b290_4c3a_9d83_98f3582fb50d.slice/crio-405477ffe1c6172ce8d6dd59e5a6f13e53d2bd8a2cd04e320e6c3cee139f5065 WatchSource:0}: Error finding container 405477ffe1c6172ce8d6dd59e5a6f13e53d2bd8a2cd04e320e6c3cee139f5065: Status 404 returned error can't find the container with id 405477ffe1c6172ce8d6dd59e5a6f13e53d2bd8a2cd04e320e6c3cee139f5065
Nov 01 23:06:41.225425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:41.225381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:41.685152 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:41.685061    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg" event={"ID":"49f56cfd-b290-4c3a-9d83-98f3582fb50d","Type":"ContainerStarted","Data":"aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66"}
Nov 01 23:06:41.685152 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:41.685112    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg" event={"ID":"49f56cfd-b290-4c3a-9d83-98f3582fb50d","Type":"ContainerStarted","Data":"405477ffe1c6172ce8d6dd59e5a6f13e53d2bd8a2cd04e320e6c3cee139f5065"}
Nov 01 23:06:41.764663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:41.764629    4531 scope.go:122] "RemoveContainer" containerID="eb3cb963c68f49477c65571173d088c6864df9b9dd0af0cebb3010162e594a4a"
Nov 01 23:06:41.777939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:41.777912    4531 scope.go:122] "RemoveContainer" containerID="a8d2cb7cc03f1ce725d5b127d1fc5285e7595e552d81a9f809aca7405d3f016c"
Nov 01 23:06:41.795004 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:41.794953    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg" podStartSLOduration=2.794940156 podStartE2EDuration="2.794940156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:06:41.792833175 +0000 UTC m=+2583.110483175" watchObservedRunningTime="2026-11-01 23:06:41.794940156 +0000 UTC m=+2583.112590157"
Nov 01 23:06:42.336152 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:42.336108    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:42.352767 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:42.352671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:42.526752 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:06:42.526707    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:06:46.029401 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:46.029358    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:46.863170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:46.863139    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:47.066467 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:47.066436    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:50.335916 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:50.335880    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:50.353930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:50.353900    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:50.574455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:50.574425    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:50.777331 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:50.777298    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:50.779493 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:50.779468    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:06:53.051483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:53.051447    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"]
Nov 01 23:06:53.097672 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:53.097636    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8617f7ac-js45h"]
Nov 01 23:06:54.816898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:54.816864    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe" path="/var/lib/kubelet/pods/fcc66b6f-a6a3-499b-98ea-e91ab6ef9cfe/volumes"
Nov 01 23:06:55.639634 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:55.639597    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:57.934516 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:57.934483    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:06:59.457996 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.457963    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"]
Nov 01 23:06:59.464658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.464634    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.514663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.514618    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:06:59.514934 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.514716    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:06:59.517006 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.516973    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"]
Nov 01 23:06:59.624274 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.624183    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.624274 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.624259    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv92w\" (UniqueName: \"kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.624535 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.624310    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.725889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.725802    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.725889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.725879    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv92w\" (UniqueName: \"kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.726127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.725924    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.726791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.726723    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.728324 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.728301    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.788499 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.788450    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv92w\" (UniqueName: \"kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w\") pod \"maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:06:59.832163 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:06:59.832118    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:07:00.221002 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:00.220916    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"]
Nov 01 23:07:00.225147 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:07:00.225112    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c6ca12e_8dec_4c5f_9fad_91c51a14e2d4.slice/crio-c3c54fd0a2767ac631a7fe1038e0b1796320b09d9048b29667f7dadbbefee26a WatchSource:0}: Error finding container c3c54fd0a2767ac631a7fe1038e0b1796320b09d9048b29667f7dadbbefee26a: Status 404 returned error can't find the container with id c3c54fd0a2767ac631a7fe1038e0b1796320b09d9048b29667f7dadbbefee26a
Nov 01 23:07:00.842114 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:00.842028    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:00.858798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:00.858762    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx" event={"ID":"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4","Type":"ContainerStarted","Data":"54a506848dfb3b6b76fbbbaa6d3096f0fdd5329104afda4e18627268d50d7340"}
Nov 01 23:07:00.858977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:00.858831    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx" event={"ID":"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4","Type":"ContainerStarted","Data":"c3c54fd0a2767ac631a7fe1038e0b1796320b09d9048b29667f7dadbbefee26a"}
Nov 01 23:07:00.980531 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:00.980468    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx" podStartSLOduration=1.980450558 podStartE2EDuration="1.980450558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:07:00.965027595 +0000 UTC m=+2602.282677602" watchObservedRunningTime="2026-11-01 23:07:00.980450558 +0000 UTC m=+2602.298100562"
Nov 01 23:07:01.870636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:01.870595    4531 generic.go:356] "Generic (PLEG): container finished" podID="5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" containerID="54a506848dfb3b6b76fbbbaa6d3096f0fdd5329104afda4e18627268d50d7340" exitCode=0
Nov 01 23:07:01.871094 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:01.870668    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx" event={"ID":"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4","Type":"ContainerDied","Data":"54a506848dfb3b6b76fbbbaa6d3096f0fdd5329104afda4e18627268d50d7340"}
Nov 01 23:07:03.025328 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.025303    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:07:03.170325 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.170247    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca\") pod \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") "
Nov 01 23:07:03.170485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.170423    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w\" (UniqueName: \"kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w\") pod \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") "
Nov 01 23:07:03.170485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.170445    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca\") pod \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\" (UID: \"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4\") "
Nov 01 23:07:03.170890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.170860    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca" pod "5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" (UID: "5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:03.172426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.172381    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca" pod "5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" (UID: "5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:07:03.172538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.172474    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w" pod "5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" (UID: "5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4"). InnerVolumeSpecName "kube-api-access-fv92w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:03.272451 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.272411    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:03.272451 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.272441    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv92w\" (UniqueName: \"kubernetes.io/projected/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-kube-api-access-fv92w\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:03.272451 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.272452    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:03.888739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.888697    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"
Nov 01 23:07:03.888921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.888761    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx" event={"ID":"5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4","Type":"ContainerDied","Data":"c3c54fd0a2767ac631a7fe1038e0b1796320b09d9048b29667f7dadbbefee26a"}
Nov 01 23:07:03.888921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:03.888809    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c54fd0a2767ac631a7fe1038e0b1796320b09d9048b29667f7dadbbefee26a"
Nov 01 23:07:05.910032 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:05.909830    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:10.643650 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:10.643614    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:11.014494 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:11.014461    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"]
Nov 01 23:07:11.122429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:11.122398    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"]
Nov 01 23:07:11.430214 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:11.430118    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:11.618823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:11.618788    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:12.546385 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:07:12.546343    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:07:15.428517 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:15.428457    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:16.014975 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.014925    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h" podUID="718b599c-fc6e-4a5b-a91b-abcefda5a3fc" containerName="payload-processing" containerID="cri-o://b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c" gracePeriod=25
Nov 01 23:07:16.123640 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.123601    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d" podUID="68ff5454-8242-403b-893e-56a69caaafa2" containerName="payload-pre-processing" containerID="cri-o://d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb" gracePeriod=25
Nov 01 23:07:16.302140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.302108    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:07:16.382910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.382886    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:07:16.443042 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.442998    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld\" (UniqueName: \"kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld\") pod \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") "
Nov 01 23:07:16.443512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443082    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc\" (UniqueName: \"kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc\") pod \"68ff5454-8242-403b-893e-56a69caaafa2\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") "
Nov 01 23:07:16.443512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443200    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume\") pod \"68ff5454-8242-403b-893e-56a69caaafa2\" (UID: \"68ff5454-8242-403b-893e-56a69caaafa2\") "
Nov 01 23:07:16.443512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443251    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume\") pod \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\" (UID: \"718b599c-fc6e-4a5b-a91b-abcefda5a3fc\") "
Nov 01 23:07:16.443703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443631    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume" pod "68ff5454-8242-403b-893e-56a69caaafa2" (UID: "68ff5454-8242-403b-893e-56a69caaafa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:16.443703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443646    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume" pod "718b599c-fc6e-4a5b-a91b-abcefda5a3fc" (UID: "718b599c-fc6e-4a5b-a91b-abcefda5a3fc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:07:16.443904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443882    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68ff5454-8242-403b-893e-56a69caaafa2-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:16.443963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.443909    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:16.445345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.445318    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc" pod "68ff5454-8242-403b-893e-56a69caaafa2" (UID: "68ff5454-8242-403b-893e-56a69caaafa2"). InnerVolumeSpecName "kube-api-access-rnmdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:16.445668 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.445650    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld" pod "718b599c-fc6e-4a5b-a91b-abcefda5a3fc" (UID: "718b599c-fc6e-4a5b-a91b-abcefda5a3fc"). InnerVolumeSpecName "kube-api-access-jttld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:07:16.545212 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.545109    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jttld\" (UniqueName: \"kubernetes.io/projected/718b599c-fc6e-4a5b-a91b-abcefda5a3fc-kube-api-access-jttld\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:16.545212 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:16.545153    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnmdc\" (UniqueName: \"kubernetes.io/projected/68ff5454-8242-403b-893e-56a69caaafa2-kube-api-access-rnmdc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:07:17.006581 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.006545    4531 generic.go:356] "Generic (PLEG): container finished" podID="718b599c-fc6e-4a5b-a91b-abcefda5a3fc" containerID="b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c" exitCode=0
Nov 01 23:07:17.006810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.006586    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"
Nov 01 23:07:17.006810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.006637    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h" event={"ID":"718b599c-fc6e-4a5b-a91b-abcefda5a3fc","Type":"ContainerDied","Data":"b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c"}
Nov 01 23:07:17.006810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.006693    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h" event={"ID":"718b599c-fc6e-4a5b-a91b-abcefda5a3fc","Type":"ContainerDied","Data":"db289fbe0b42ee5d78b5e121febbd57688f5b39b94ca7913c05c1e2a23576f15"}
Nov 01 23:07:17.006810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.006722    4531 scope.go:122] "RemoveContainer" containerID="b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c"
Nov 01 23:07:17.008397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.008368    4531 generic.go:356] "Generic (PLEG): container finished" podID="68ff5454-8242-403b-893e-56a69caaafa2" containerID="d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb" exitCode=0
Nov 01 23:07:17.008516 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.008418    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"
Nov 01 23:07:17.008607 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.008588    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d" event={"ID":"68ff5454-8242-403b-893e-56a69caaafa2","Type":"ContainerDied","Data":"d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb"}
Nov 01 23:07:17.008676 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.008631    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d" event={"ID":"68ff5454-8242-403b-893e-56a69caaafa2","Type":"ContainerDied","Data":"2f0da0459e692df83ae54e6a3ef27901c009b116acf189febad56ba5e979ef00"}
Nov 01 23:07:17.017858 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.017661    4531 scope.go:122] "RemoveContainer" containerID="b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c"
Nov 01 23:07:17.017974 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:07:17.017956    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c\": container with ID starting with b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c not found: ID does not exist" containerID="b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c"
Nov 01 23:07:17.018019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.017980    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c"} err="failed to get container status \"b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c\": rpc error: code = NotFound desc = could not find container \"b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c\": container with ID starting with b9e50b8f1f733488010e2dba536ffa308a52a4f8f986f0b609eba3861cf5241c not found: ID does not exist"
Nov 01 23:07:17.018019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.017997    4531 scope.go:122] "RemoveContainer" containerID="d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb"
Nov 01 23:07:17.037889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.037873    4531 scope.go:122] "RemoveContainer" containerID="d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb"
Nov 01 23:07:17.038145 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:07:17.038127    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb\": container with ID starting with d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb not found: ID does not exist" containerID="d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb"
Nov 01 23:07:17.038191 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.038151    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb"} err="failed to get container status \"d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb\": rpc error: code = NotFound desc = could not find container \"d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb\": container with ID starting with d4478d19e06b2a4be9be0cf90ed665cc856f3055047ef3dbf461a0d97b31defb not found: ID does not exist"
Nov 01 23:07:17.170169 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.170112    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"]
Nov 01 23:07:17.216568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.216529    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-55b58e-884cf95dc-6jj6h"]
Nov 01 23:07:17.322406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.322374    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"]
Nov 01 23:07:17.326035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:17.326008    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-55b58e-698746948c-8258d"]
Nov 01 23:07:18.818092 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:18.818064    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ff5454-8242-403b-893e-56a69caaafa2" path="/var/lib/kubelet/pods/68ff5454-8242-403b-893e-56a69caaafa2/volumes"
Nov 01 23:07:18.818494 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:18.818485    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718b599c-fc6e-4a5b-a91b-abcefda5a3fc" path="/var/lib/kubelet/pods/718b599c-fc6e-4a5b-a91b-abcefda5a3fc/volumes"
Nov 01 23:07:20.247009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:20.246969    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:25.054489 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:25.054449    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:29.836819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:29.836787    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:34.630320 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:34.630239    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:39.370471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:39.370440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:41.834161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:41.834129    4531 scope.go:122] "RemoveContainer" containerID="a23ee382d2da97689dddb5cba239e2a3f75c6c320fc86c5985454e0895b6d721"
Nov 01 23:07:41.844542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:41.844525    4531 scope.go:122] "RemoveContainer" containerID="b1e6b3997eb7c51a73df6546e16c470584f44c0c24d7c14b6ac56fff64280ef8"
Nov 01 23:07:41.854075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:41.854056    4531 scope.go:122] "RemoveContainer" containerID="7f24663084b5f4be0d73f2215f60a322ae7c296f04d9b8d5baa23c837c8fd196"
Nov 01 23:07:41.863238 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:41.863220    4531 scope.go:122] "RemoveContainer" containerID="e9c428d95eb05d8abbd598b43ea6a20654a0d5cdb05c7c282ee5a4e45d8209f8"
Nov 01 23:07:42.528150 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:07:42.528114    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:07:42.724210 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:42.724181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:44.238272 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:44.238233    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:49.031151 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:49.031071    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:53.932081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:53.932044    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:07:58.788026 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:07:58.787986    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:03.512816 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:03.512780    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:08.392409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:08.392373    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:12.527060 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:08:12.527018    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:08:13.122021 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:13.121991    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:17.933946 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:17.933875    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:18.128295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:18.128261    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:19.071920 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:19.071888    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"]
Nov 01 23:08:19.118045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:19.118014    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c2c73b2-dnltz"]
Nov 01 23:08:20.048830 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.048793    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"]
Nov 01 23:08:20.094298 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.094264    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"]
Nov 01 23:08:20.097815 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.097793    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"]
Nov 01 23:08:20.100837 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.100817    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b616f3e1-4clbc"]
Nov 01 23:08:20.103757 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.103735    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ccdcf4d7-lw75w"]
Nov 01 23:08:20.139946 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.139923    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27032c97-xfpct"]
Nov 01 23:08:20.818543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.818504    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350" path="/var/lib/kubelet/pods/7d9c0bb4-9bf3-473c-9d19-cea9ce4b1350/volumes"
Nov 01 23:08:20.819132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.819115    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1" path="/var/lib/kubelet/pods/a1ba1cdc-ddec-4f10-b1a4-5c4b07d33ba1/volumes"
Nov 01 23:08:20.819636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.819621    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6326600-cdb1-4a78-a51a-dc3dcb76230c" path="/var/lib/kubelet/pods/f6326600-cdb1-4a78-a51a-dc3dcb76230c/volumes"
Nov 01 23:08:20.820699 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:20.820679    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9" path="/var/lib/kubelet/pods/ff4f78aa-0ddc-4b18-8d09-a88ed28e61b9/volumes"
Nov 01 23:08:22.745219 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:22.745181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:27.536670 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:27.536619    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:31.992966 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:31.992928    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:32.173049 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:32.173019    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:32.353154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:32.353070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:37.124402 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:37.124359    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:41.841970 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:41.841882    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:41.916918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:41.916891    4531 scope.go:122] "RemoveContainer" containerID="b0b231fdcae44bbe4f53f0abca1955ea39e9111d00590e5f8c6248e2682e40a3"
Nov 01 23:08:41.927252 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:41.927225    4531 scope.go:122] "RemoveContainer" containerID="42391f02ce417ce978dca0cdadef0ae04db51d79c0b4c5d3fede71cdf8f2f3fb"
Nov 01 23:08:42.526701 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:08:42.526668    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:08:43.036900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:43.036862    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:46.603202 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:46.603168    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:50.660592 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:50.660562    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"]
Nov 01 23:08:50.661084 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:50.660842    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" podUID="8b2d55f6-2ac2-4343-a801-31e2e3b50e57" containerName="maas-api" containerID="cri-o://580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492" gracePeriod=30
Nov 01 23:08:50.913566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:50.913494    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:51.026070 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.025719    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:08:51.188755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.188670    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\") pod \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") "
Nov 01 23:08:51.188755 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.188713    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7\" (UniqueName: \"kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7\") pod \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") "
Nov 01 23:08:51.189008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.188766    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs\") pod \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") "
Nov 01 23:08:51.189008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.188891    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca\") pod \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\" (UID: \"8b2d55f6-2ac2-4343-a801-31e2e3b50e57\") "
Nov 01 23:08:51.189311 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.189282    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca" pod "8b2d55f6-2ac2-4343-a801-31e2e3b50e57" (UID: "8b2d55f6-2ac2-4343-a801-31e2e3b50e57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:08:51.189438 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.189413    4531 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-openshift-service-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:08:51.191058 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.191028    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs" pod "8b2d55f6-2ac2-4343-a801-31e2e3b50e57" (UID: "8b2d55f6-2ac2-4343-a801-31e2e3b50e57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:08:51.191058 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.191043    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls" pod "8b2d55f6-2ac2-4343-a801-31e2e3b50e57" (UID: "8b2d55f6-2ac2-4343-a801-31e2e3b50e57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:08:51.191275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.191259    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7" pod "8b2d55f6-2ac2-4343-a801-31e2e3b50e57" (UID: "8b2d55f6-2ac2-4343-a801-31e2e3b50e57"). InnerVolumeSpecName "kube-api-access-rzgj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:08:51.290395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.290367    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-maas-api-tls\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:08:51.290395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.290395    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgj7\" (UniqueName: \"kubernetes.io/projected/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-kube-api-access-rzgj7\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:08:51.290557 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.290406    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b2d55f6-2ac2-4343-a801-31e2e3b50e57-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:08:51.789443 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.789404    4531 generic.go:356] "Generic (PLEG): container finished" podID="8b2d55f6-2ac2-4343-a801-31e2e3b50e57" containerID="580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492" exitCode=0
Nov 01 23:08:51.789904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.789449    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"
Nov 01 23:08:51.789904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.789491    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" event={"ID":"8b2d55f6-2ac2-4343-a801-31e2e3b50e57","Type":"ContainerDied","Data":"580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492"}
Nov 01 23:08:51.789904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.789567    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk" event={"ID":"8b2d55f6-2ac2-4343-a801-31e2e3b50e57","Type":"ContainerDied","Data":"4a73cf522f53b016eb29124be350ef7c93552f5291009775b6bbadee30203b9c"}
Nov 01 23:08:51.789904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.789603    4531 scope.go:122] "RemoveContainer" containerID="580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492"
Nov 01 23:08:51.801424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.801405    4531 scope.go:122] "RemoveContainer" containerID="580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492"
Nov 01 23:08:51.801758 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:08:51.801739    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492\": container with ID starting with 580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492 not found: ID does not exist" containerID="580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492"
Nov 01 23:08:51.801818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.801763    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492"} err="failed to get container status \"580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492\": rpc error: code = NotFound desc = could not find container \"580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492\": container with ID starting with 580475bf7fc9f82b0f948995f166d045c85e46aa9543140c37c7855f574b5492 not found: ID does not exist"
Nov 01 23:08:51.946078 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.946040    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"]
Nov 01 23:08:51.992390 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:51.992357    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-55b58e-7f55c67d89-kvdqk"]
Nov 01 23:08:52.309047 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:52.309004    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:52.533608 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:52.533569    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:52.816654 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:52.816622    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b2d55f6-2ac2-4343-a801-31e2e3b50e57" path="/var/lib/kubelet/pods/8b2d55f6-2ac2-4343-a801-31e2e3b50e57/volumes"
Nov 01 23:08:52.905712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:52.905681    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:52.956776 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:52.956744    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"]
Nov 01 23:08:53.005295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:53.005265    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-55b58e-9mvjx"]
Nov 01 23:08:53.457964 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:53.457931    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:54.817538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:54.817500    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" path="/var/lib/kubelet/pods/5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4/volumes"
Nov 01 23:08:57.701488 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:57.701454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:08:59.878773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:08:59.878740    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:00.094270 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.091201    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:00.284347 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.284310    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:00.991924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.991892    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"]
Nov 01 23:09:00.992460 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992446    4531 cpu_manager.go:414] "containerMap: removing container" podUID="718b599c-fc6e-4a5b-a91b-abcefda5a3fc" containerName="payload-processing"
Nov 01 23:09:00.992512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992460    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718b599c-fc6e-4a5b-a91b-abcefda5a3fc" containerName="payload-processing"
Nov 01 23:09:00.992512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992477    4531 cpu_manager.go:414] "containerMap: removing container" podUID="68ff5454-8242-403b-893e-56a69caaafa2" containerName="payload-pre-processing"
Nov 01 23:09:00.992512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992482    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ff5454-8242-403b-893e-56a69caaafa2" containerName="payload-pre-processing"
Nov 01 23:09:00.992512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992506    4531 cpu_manager.go:414] "containerMap: removing container" podUID="5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" containerName="revoke-keys"
Nov 01 23:09:00.992512 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992510    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" containerName="revoke-keys"
Nov 01 23:09:00.992677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992525    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8b2d55f6-2ac2-4343-a801-31e2e3b50e57" containerName="maas-api"
Nov 01 23:09:00.992677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992530    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b2d55f6-2ac2-4343-a801-31e2e3b50e57" containerName="maas-api"
Nov 01 23:09:00.992677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992608    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="718b599c-fc6e-4a5b-a91b-abcefda5a3fc" containerName="payload-processing"
Nov 01 23:09:00.992677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992617    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ff5454-8242-403b-893e-56a69caaafa2" containerName="payload-pre-processing"
Nov 01 23:09:00.992677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992628    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c6ca12e-8dec-4c5f-9fad-91c51a14e2d4" containerName="revoke-keys"
Nov 01 23:09:00.992677 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.992636    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b2d55f6-2ac2-4343-a801-31e2e3b50e57" containerName="maas-api"
Nov 01 23:09:00.997486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:00.997466    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.005068 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.005034    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.005196 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.005089    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.005279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.005246    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztsfv\" (UniqueName: \"kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.046433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.046406    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:09:01.092997 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.092965    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:09:01.096738 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.096704    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"]
Nov 01 23:09:01.106376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.106356    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztsfv\" (UniqueName: \"kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.106510 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.106491    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.106562 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.106529    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.107145 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.107120    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.108850 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.108832    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.159986 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.159967    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztsfv\" (UniqueName: \"kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv\") pod \"maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.312852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.312793    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:01.689852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.689781    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"]
Nov 01 23:09:01.693300 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:09:01.693269    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d92bd53_1bec_498f_b5c2_55e5868f9a75.slice/crio-4546d8ed6e3d721a265245f6258543edbc783ab1681fa461878d2d8319bd3286 WatchSource:0}: Error finding container 4546d8ed6e3d721a265245f6258543edbc783ab1681fa461878d2d8319bd3286: Status 404 returned error can't find the container with id 4546d8ed6e3d721a265245f6258543edbc783ab1681fa461878d2d8319bd3286
Nov 01 23:09:01.871434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:01.871400    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6" event={"ID":"2d92bd53-1bec-498f-b5c2-55e5868f9a75","Type":"ContainerStarted","Data":"4546d8ed6e3d721a265245f6258543edbc783ab1681fa461878d2d8319bd3286"}
Nov 01 23:09:02.845488 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:02.845409    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:02.897743 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:02.896825    4531 generic.go:356] "Generic (PLEG): container finished" podID="2d92bd53-1bec-498f-b5c2-55e5868f9a75" containerID="b464b290a486bff61fc420d384a1e99a2d92d05f83da49616a1de598ef28f5cc" exitCode=0
Nov 01 23:09:02.897743 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:02.896872    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6" event={"ID":"2d92bd53-1bec-498f-b5c2-55e5868f9a75","Type":"ContainerDied","Data":"b464b290a486bff61fc420d384a1e99a2d92d05f83da49616a1de598ef28f5cc"}
Nov 01 23:09:04.049841 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.049817    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:04.140475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.140445    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca\") pod \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") "
Nov 01 23:09:04.140666 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.140646    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv\" (UniqueName: \"kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv\") pod \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") "
Nov 01 23:09:04.140765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.140687    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca\") pod \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\" (UID: \"2d92bd53-1bec-498f-b5c2-55e5868f9a75\") "
Nov 01 23:09:04.141127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.141092    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca" pod "2d92bd53-1bec-498f-b5c2-55e5868f9a75" (UID: "2d92bd53-1bec-498f-b5c2-55e5868f9a75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:04.142626 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.142601    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca" pod "2d92bd53-1bec-498f-b5c2-55e5868f9a75" (UID: "2d92bd53-1bec-498f-b5c2-55e5868f9a75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:09:04.142840 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.142818    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv" pod "2d92bd53-1bec-498f-b5c2-55e5868f9a75" (UID: "2d92bd53-1bec-498f-b5c2-55e5868f9a75"). InnerVolumeSpecName "kube-api-access-ztsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:04.242819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.242781    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztsfv\" (UniqueName: \"kubernetes.io/projected/2d92bd53-1bec-498f-b5c2-55e5868f9a75-kube-api-access-ztsfv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:04.242819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.242813    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:04.242819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.242824    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d92bd53-1bec-498f-b5c2-55e5868f9a75-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:04.916838 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.916812    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"
Nov 01 23:09:04.917009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.916853    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6" event={"ID":"2d92bd53-1bec-498f-b5c2-55e5868f9a75","Type":"ContainerDied","Data":"4546d8ed6e3d721a265245f6258543edbc783ab1681fa461878d2d8319bd3286"}
Nov 01 23:09:04.917009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:04.916899    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4546d8ed6e3d721a265245f6258543edbc783ab1681fa461878d2d8319bd3286"
Nov 01 23:09:07.922456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:07.922414    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:12.527919 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:09:12.527877    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:09:12.983749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:12.979845    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:13.312685 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:13.312590    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:17.644798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:17.644758    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:22.449538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:22.449459    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:27.230765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:27.230700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:32.038288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:32.038246    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:36.764634 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.764542    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:36.908235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.908205    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"]
Nov 01 23:09:36.908839 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.908825    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d92bd53-1bec-498f-b5c2-55e5868f9a75" containerName="revoke-keys"
Nov 01 23:09:36.908897 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.908840    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d92bd53-1bec-498f-b5c2-55e5868f9a75" containerName="revoke-keys"
Nov 01 23:09:36.908950 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.908941    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d92bd53-1bec-498f-b5c2-55e5868f9a75" containerName="revoke-keys"
Nov 01 23:09:36.912357 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.912337    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:36.969643 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:36.969614    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"]
Nov 01 23:09:37.010578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.010555    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:09:37.010705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.010636    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:09:37.015879 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.015832    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.015970 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.015955    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.016019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.015983    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9x2l\" (UniqueName: \"kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.116913 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.116879    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.117122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.116969    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.117122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.116991    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9x2l\" (UniqueName: \"kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.117641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.117614    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.119336 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.119310    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.171080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.171048    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9x2l\" (UniqueName: \"kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l\") pod \"maas-api-revoke-sub-782ff25b-4tr4m\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.225845 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.225820    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:37.601046 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:37.600978    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"]
Nov 01 23:09:37.604392 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:09:37.604359    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8ad203_051b_4a48_a659_3497e2c83883.slice/crio-d7bba8c793c1b52f3b8c6733d77ec7265923eed1453bee9c771c2fc2929728bc WatchSource:0}: Error finding container d7bba8c793c1b52f3b8c6733d77ec7265923eed1453bee9c771c2fc2929728bc: Status 404 returned error can't find the container with id d7bba8c793c1b52f3b8c6733d77ec7265923eed1453bee9c771c2fc2929728bc
Nov 01 23:09:38.179035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:38.178945    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m" event={"ID":"4a8ad203-051b-4a48-a659-3497e2c83883","Type":"ContainerStarted","Data":"0243c85fa1ead2e2bf5c210f3d9c987e32a612c397c6677361baf0eb45c005c0"}
Nov 01 23:09:38.179035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:38.179020    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m" event={"ID":"4a8ad203-051b-4a48-a659-3497e2c83883","Type":"ContainerStarted","Data":"d7bba8c793c1b52f3b8c6733d77ec7265923eed1453bee9c771c2fc2929728bc"}
Nov 01 23:09:38.296250 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:38.296195    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m" podStartSLOduration=2.296182506 podStartE2EDuration="2.296182506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:38.292138176 +0000 UTC m=+2759.609788189" watchObservedRunningTime="2026-11-01 23:09:38.296182506 +0000 UTC m=+2759.613832500"
Nov 01 23:09:39.187627 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:39.187596    4531 generic.go:356] "Generic (PLEG): container finished" podID="4a8ad203-051b-4a48-a659-3497e2c83883" containerID="0243c85fa1ead2e2bf5c210f3d9c987e32a612c397c6677361baf0eb45c005c0" exitCode=0
Nov 01 23:09:39.188021 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:39.187679    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m" event={"ID":"4a8ad203-051b-4a48-a659-3497e2c83883","Type":"ContainerDied","Data":"0243c85fa1ead2e2bf5c210f3d9c987e32a612c397c6677361baf0eb45c005c0"}
Nov 01 23:09:40.366614 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.366588    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:40.452360 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.452294    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca\") pod \"4a8ad203-051b-4a48-a659-3497e2c83883\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") "
Nov 01 23:09:40.452506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.452396    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca\") pod \"4a8ad203-051b-4a48-a659-3497e2c83883\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") "
Nov 01 23:09:40.452564 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.452528    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l\" (UniqueName: \"kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l\") pod \"4a8ad203-051b-4a48-a659-3497e2c83883\" (UID: \"4a8ad203-051b-4a48-a659-3497e2c83883\") "
Nov 01 23:09:40.452633 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.452608    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca" pod "4a8ad203-051b-4a48-a659-3497e2c83883" (UID: "4a8ad203-051b-4a48-a659-3497e2c83883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:40.452896 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.452878    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:40.454504 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.454483    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l" pod "4a8ad203-051b-4a48-a659-3497e2c83883" (UID: "4a8ad203-051b-4a48-a659-3497e2c83883"). InnerVolumeSpecName "kube-api-access-f9x2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:40.454602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.454584    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca" pod "4a8ad203-051b-4a48-a659-3497e2c83883" (UID: "4a8ad203-051b-4a48-a659-3497e2c83883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:09:40.553794 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.553764    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9x2l\" (UniqueName: \"kubernetes.io/projected/4a8ad203-051b-4a48-a659-3497e2c83883-kube-api-access-f9x2l\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:40.553794 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:40.553792    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a8ad203-051b-4a48-a659-3497e2c83883-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:41.206156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.206116    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"
Nov 01 23:09:41.206156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.206127    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m" event={"ID":"4a8ad203-051b-4a48-a659-3497e2c83883","Type":"ContainerDied","Data":"d7bba8c793c1b52f3b8c6733d77ec7265923eed1453bee9c771c2fc2929728bc"}
Nov 01 23:09:41.206387 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.206214    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7bba8c793c1b52f3b8c6733d77ec7265923eed1453bee9c771c2fc2929728bc"
Nov 01 23:09:41.634781 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.634664    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:41.964063 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.964030    4531 scope.go:122] "RemoveContainer" containerID="7898789f9312cd7dc4ee30d4ad101c345e1651434431ebfbb2ae50672dfdbd65"
Nov 01 23:09:41.983229 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.983204    4531 scope.go:122] "RemoveContainer" containerID="4251befbc7237ecf994649ffeba62e82aff1a1d68f1422baa5134c83a46c01bd"
Nov 01 23:09:41.995061 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:41.995037    4531 scope.go:122] "RemoveContainer" containerID="451766247f76e08ee5e417bf4b895de3b445755482e8981805fe2519ad6e0bf2"
Nov 01 23:09:42.005876 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:42.005850    4531 scope.go:122] "RemoveContainer" containerID="1dcd906697a83448b4c6d10f9edc9f35029ad84a30749b0cdd35d865d9aadf26"
Nov 01 23:09:42.014566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:42.014544    4531 scope.go:122] "RemoveContainer" containerID="857c75ed8bf41a23be3b90f0253736970a385c834f6a14f9ccf00beaab2915f1"
Nov 01 23:09:42.023154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:42.023137    4531 scope.go:122] "RemoveContainer" containerID="6b143d64a2eb51a39aa6159fd2f8ef2bfd19a44c9830beeb2a87009bc4a2446d"
Nov 01 23:09:42.527918 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:09:42.527883    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:09:43.376613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:43.376568    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:46.416220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:46.416153    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:48.858315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:48.858278    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"]
Nov 01 23:09:48.943959 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:48.942802    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:49.848120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.848086    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"]
Nov 01 23:09:49.848652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.848639    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4a8ad203-051b-4a48-a659-3497e2c83883" containerName="revoke-keys"
Nov 01 23:09:49.848705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.848653    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a8ad203-051b-4a48-a659-3497e2c83883" containerName="revoke-keys"
Nov 01 23:09:49.848800 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.848789    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a8ad203-051b-4a48-a659-3497e2c83883" containerName="revoke-keys"
Nov 01 23:09:49.853990 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.853970    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:49.908476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.908448    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"]
Nov 01 23:09:49.948781 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.948756    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:09:49.948931 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.948815    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:09:49.965235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.965207    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:49.965422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.965347    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24pms\" (UniqueName: \"kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:49.965497 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:49.965467    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.066266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.066240    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.066429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.066277    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24pms\" (UniqueName: \"kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.066429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.066333    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.067137 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.067113    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.068611 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.068587    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.120398 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.120344    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24pms\" (UniqueName: \"kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms\") pod \"maas-api-revoke-sub-f3ee9e5a-mv7vq\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.167946 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.167916    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:50.545653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:50.545623    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"]
Nov 01 23:09:50.549512 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:09:50.549481    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea78909a_3497_4179_8f45_7c6221b15172.slice/crio-e7d1ebb81ffacfe09c9819e6726fd196eb908b3ba390fd80c9e31826a2ddb7ce WatchSource:0}: Error finding container e7d1ebb81ffacfe09c9819e6726fd196eb908b3ba390fd80c9e31826a2ddb7ce: Status 404 returned error can't find the container with id e7d1ebb81ffacfe09c9819e6726fd196eb908b3ba390fd80c9e31826a2ddb7ce
Nov 01 23:09:51.284662 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:51.284566    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq" event={"ID":"ea78909a-3497-4179-8f45-7c6221b15172","Type":"ContainerStarted","Data":"505e74e8c69b78d028f692af7d340b06f52111893e63707014c6d1fb2ddac496"}
Nov 01 23:09:51.284662 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:51.284626    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq" event={"ID":"ea78909a-3497-4179-8f45-7c6221b15172","Type":"ContainerStarted","Data":"e7d1ebb81ffacfe09c9819e6726fd196eb908b3ba390fd80c9e31826a2ddb7ce"}
Nov 01 23:09:51.395861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:51.395805    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq" podStartSLOduration=2.395790852 podStartE2EDuration="2.395790852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:09:51.392670512 +0000 UTC m=+2772.710320514" watchObservedRunningTime="2026-11-01 23:09:51.395790852 +0000 UTC m=+2772.713440899"
Nov 01 23:09:51.435578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:51.435547    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:51.911049 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:51.911014    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"]
Nov 01 23:09:51.911308 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:51.911278    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" containerID="cri-o://427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c" gracePeriod=30
Nov 01 23:09:52.296995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:52.296964    4531 generic.go:356] "Generic (PLEG): container finished" podID="ea78909a-3497-4179-8f45-7c6221b15172" containerID="505e74e8c69b78d028f692af7d340b06f52111893e63707014c6d1fb2ddac496" exitCode=0
Nov 01 23:09:52.297537 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:52.297048    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq" event={"ID":"ea78909a-3497-4179-8f45-7c6221b15172","Type":"ContainerDied","Data":"505e74e8c69b78d028f692af7d340b06f52111893e63707014c6d1fb2ddac496"}
Nov 01 23:09:53.024746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.024700    4531 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 23:09:53.024918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.024771    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 23:09:53.462247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.462219    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:53.624877 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.624813    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca\") pod \"ea78909a-3497-4179-8f45-7c6221b15172\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") "
Nov 01 23:09:53.625016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.624929    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca\") pod \"ea78909a-3497-4179-8f45-7c6221b15172\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") "
Nov 01 23:09:53.625082 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.625062    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms\" (UniqueName: \"kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms\") pod \"ea78909a-3497-4179-8f45-7c6221b15172\" (UID: \"ea78909a-3497-4179-8f45-7c6221b15172\") "
Nov 01 23:09:53.625155 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.625133    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca" pod "ea78909a-3497-4179-8f45-7c6221b15172" (UID: "ea78909a-3497-4179-8f45-7c6221b15172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:53.625497 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.625476    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:53.627069 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.627045    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca" pod "ea78909a-3497-4179-8f45-7c6221b15172" (UID: "ea78909a-3497-4179-8f45-7c6221b15172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:09:53.627195 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.627158    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms" pod "ea78909a-3497-4179-8f45-7c6221b15172" (UID: "ea78909a-3497-4179-8f45-7c6221b15172"). InnerVolumeSpecName "kube-api-access-24pms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:53.726807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.726779    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24pms\" (UniqueName: \"kubernetes.io/projected/ea78909a-3497-4179-8f45-7c6221b15172-kube-api-access-24pms\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:53.726807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:53.726806    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea78909a-3497-4179-8f45-7c6221b15172-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:54.313605 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:54.313575    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"
Nov 01 23:09:54.313799 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:54.313613    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq" event={"ID":"ea78909a-3497-4179-8f45-7c6221b15172","Type":"ContainerDied","Data":"e7d1ebb81ffacfe09c9819e6726fd196eb908b3ba390fd80c9e31826a2ddb7ce"}
Nov 01 23:09:54.313799 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:54.313661    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d1ebb81ffacfe09c9819e6726fd196eb908b3ba390fd80c9e31826a2ddb7ce"
Nov 01 23:09:56.193533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:56.193496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:09:57.054377 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.054344    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"]
Nov 01 23:09:57.102748 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.100247    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0a4fbfd-xqfhv"]
Nov 01 23:09:57.112384 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.112353    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"]
Nov 01 23:09:57.118359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.118331    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15353b44-r8lx9"]
Nov 01 23:09:57.179938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.179911    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 23:09:57.262556 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262528    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262568    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262600    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262652    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262764    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262826    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262895    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262905    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262923    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:57.262995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.262951    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.263491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263021    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.263491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263141    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:09:57.263491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263150    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token\") pod \"72da0505-3e3f-4764-a695-faa46ae5a8d3\" (UID: \"72da0505-3e3f-4764-a695-faa46ae5a8d3\") "
Nov 01 23:09:57.263491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263217    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:57.263803 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263781    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72da0505-3e3f-4764-a695-faa46ae5a8d3-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.263884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263803    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.263884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263813    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.263884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263822    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.263884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.263832    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.265097 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.265076    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:09:57.265177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.265121    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:57.265390 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.265364    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:09:57.265390 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.265375    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s" pod "72da0505-3e3f-4764-a695-faa46ae5a8d3" (UID: "72da0505-3e3f-4764-a695-faa46ae5a8d3"). InnerVolumeSpecName "kube-api-access-c9r8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:09:57.337988 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.337957    4531 generic.go:356] "Generic (PLEG): container finished" podID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerID="427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c" exitCode=0
Nov 01 23:09:57.338141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.338005    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"
Nov 01 23:09:57.338141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.338042    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" event={"ID":"72da0505-3e3f-4764-a695-faa46ae5a8d3","Type":"ContainerDied","Data":"427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c"}
Nov 01 23:09:57.338141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.338099    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk" event={"ID":"72da0505-3e3f-4764-a695-faa46ae5a8d3","Type":"ContainerDied","Data":"79537ed3ea844aed38c882d548ad59a11c546baf6e4f05fed6987e2019e99e03"}
Nov 01 23:09:57.338141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.338128    4531 scope.go:122] "RemoveContainer" containerID="427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c"
Nov 01 23:09:57.350433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.350414    4531 scope.go:122] "RemoveContainer" containerID="427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c"
Nov 01 23:09:57.350698 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:09:57.350679    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c\": container with ID starting with 427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c not found: ID does not exist" containerID="427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c"
Nov 01 23:09:57.350791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.350708    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c"} err="failed to get container status \"427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c\": rpc error: code = NotFound desc = could not find container \"427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c\": container with ID starting with 427bd244d64f9da41e0ee2a2cf862fbb7356c78169c1396acf78be8af37ce98c not found: ID does not exist"
Nov 01 23:09:57.366105 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.366081    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.366105 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.366102    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.366233 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.366111    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72da0505-3e3f-4764-a695-faa46ae5a8d3-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.366233 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.366125    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9r8s\" (UniqueName: \"kubernetes.io/projected/72da0505-3e3f-4764-a695-faa46ae5a8d3-kube-api-access-c9r8s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:09:57.500956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.500919    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"]
Nov 01 23:09:57.508224 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:57.508188    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8bac2f-openshift-default-66495cf699-fwmkk"]
Nov 01 23:09:58.818890 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:58.818854    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6340ce-216c-4d41-a580-dc613c28e826" path="/var/lib/kubelet/pods/2e6340ce-216c-4d41-a580-dc613c28e826/volumes"
Nov 01 23:09:58.819356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:58.819340    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" path="/var/lib/kubelet/pods/72da0505-3e3f-4764-a695-faa46ae5a8d3/volumes"
Nov 01 23:09:58.819751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:09:58.819738    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a9a915-755c-49ef-a807-0ecf172ab418" path="/var/lib/kubelet/pods/e6a9a915-755c-49ef-a807-0ecf172ab418/volumes"
Nov 01 23:10:01.069179 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:01.069137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:03.879799 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:03.879709    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6" podUID="1bc54405-3858-4828-8b79-8c0b35bc40f8" containerName="main" containerID="cri-o://25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c" gracePeriod=45
Nov 01 23:10:04.149654 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.149627    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:10:04.240765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.240720    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache\") pod \"1bc54405-3858-4828-8b79-8c0b35bc40f8\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") "
Nov 01 23:10:04.240957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.240786    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir\") pod \"1bc54405-3858-4828-8b79-8c0b35bc40f8\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") "
Nov 01 23:10:04.240957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.240832    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm\") pod \"1bc54405-3858-4828-8b79-8c0b35bc40f8\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") "
Nov 01 23:10:04.240957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.240918    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn\" (UniqueName: \"kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn\") pod \"1bc54405-3858-4828-8b79-8c0b35bc40f8\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") "
Nov 01 23:10:04.240957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.240953    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs\") pod \"1bc54405-3858-4828-8b79-8c0b35bc40f8\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") "
Nov 01 23:10:04.241170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241001    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home\") pod \"1bc54405-3858-4828-8b79-8c0b35bc40f8\" (UID: \"1bc54405-3858-4828-8b79-8c0b35bc40f8\") "
Nov 01 23:10:04.241170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241024    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache" pod "1bc54405-3858-4828-8b79-8c0b35bc40f8" (UID: "1bc54405-3858-4828-8b79-8c0b35bc40f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:04.241170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241070    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir" pod "1bc54405-3858-4828-8b79-8c0b35bc40f8" (UID: "1bc54405-3858-4828-8b79-8c0b35bc40f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:04.241331 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241300    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home" pod "1bc54405-3858-4828-8b79-8c0b35bc40f8" (UID: "1bc54405-3858-4828-8b79-8c0b35bc40f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:04.241495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241477    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:04.241495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241495    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:04.241645 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.241505    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:04.243288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.243256    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn" pod "1bc54405-3858-4828-8b79-8c0b35bc40f8" (UID: "1bc54405-3858-4828-8b79-8c0b35bc40f8"). InnerVolumeSpecName "kube-api-access-jxvpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:04.243288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.243259    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs" pod "1bc54405-3858-4828-8b79-8c0b35bc40f8" (UID: "1bc54405-3858-4828-8b79-8c0b35bc40f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:10:04.243416 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.243266    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm" pod "1bc54405-3858-4828-8b79-8c0b35bc40f8" (UID: "1bc54405-3858-4828-8b79-8c0b35bc40f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:04.342579 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.342532    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxvpn\" (UniqueName: \"kubernetes.io/projected/1bc54405-3858-4828-8b79-8c0b35bc40f8-kube-api-access-jxvpn\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:04.342579 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.342573    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bc54405-3858-4828-8b79-8c0b35bc40f8-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:04.342579 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.342584    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bc54405-3858-4828-8b79-8c0b35bc40f8-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:04.395809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.395708    4531 generic.go:356] "Generic (PLEG): container finished" podID="1bc54405-3858-4828-8b79-8c0b35bc40f8" containerID="25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c" exitCode=0
Nov 01 23:10:04.395809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.395774    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"
Nov 01 23:10:04.395809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.395791    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6" event={"ID":"1bc54405-3858-4828-8b79-8c0b35bc40f8","Type":"ContainerDied","Data":"25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c"}
Nov 01 23:10:04.396108 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.395844    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6" event={"ID":"1bc54405-3858-4828-8b79-8c0b35bc40f8","Type":"ContainerDied","Data":"120e92ae6acb09453ea221ad5a3f898d50b2d23ab57b661f13d06b3550de15a9"}
Nov 01 23:10:04.396108 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.395869    4531 scope.go:122] "RemoveContainer" containerID="25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c"
Nov 01 23:10:04.407102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.407083    4531 scope.go:122] "RemoveContainer" containerID="25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c"
Nov 01 23:10:04.407397 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:10:04.407371    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c\": container with ID starting with 25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c not found: ID does not exist" containerID="25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c"
Nov 01 23:10:04.407478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.407409    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c"} err="failed to get container status \"25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c\": rpc error: code = NotFound desc = could not find container \"25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c\": container with ID starting with 25e06ce6611afa4cd7c5b60f829c8c281a593bada784faddbb1c89b11ad4885c not found: ID does not exist"
Nov 01 23:10:04.552947 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.552909    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"]
Nov 01 23:10:04.557659 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.557631    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-675d1d/test-model-675d1d-kserve-5867f55b49-klts6"]
Nov 01 23:10:04.817778 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:04.817710    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bc54405-3858-4828-8b79-8c0b35bc40f8" path="/var/lib/kubelet/pods/1bc54405-3858-4828-8b79-8c0b35bc40f8/volumes"
Nov 01 23:10:05.919716 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:05.919680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:08.208287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:08.208246    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"]
Nov 01 23:10:08.208738 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:08.208507    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy" containerID="cri-o://4a9de364354d66c7710414134167b2bfda9c1cbbdd0e4f0e39f750c8e7cc10fe" gracePeriod=30
Nov 01 23:10:08.254063 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:08.254031    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"]
Nov 01 23:10:08.254442 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:08.254410    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy" containerID="cri-o://7b0d6dd705674572a99557d4d6679b1610110ba2a328da4926520f1500257b4b" gracePeriod=30
Nov 01 23:10:10.746081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:10.746024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:12.525677 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:10:12.525639    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:10:13.477387 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.477309    4531 generic.go:356] "Generic (PLEG): container finished" podID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerID="7b0d6dd705674572a99557d4d6679b1610110ba2a328da4926520f1500257b4b" exitCode=0
Nov 01 23:10:13.477558 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.477475    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" event={"ID":"80b7dd35-ea1b-4285-88cf-fc5b404039f3","Type":"ContainerDied","Data":"7b0d6dd705674572a99557d4d6679b1610110ba2a328da4926520f1500257b4b"}
Nov 01 23:10:13.479673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.479646    4531 generic.go:356] "Generic (PLEG): container finished" podID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerID="4a9de364354d66c7710414134167b2bfda9c1cbbdd0e4f0e39f750c8e7cc10fe" exitCode=0
Nov 01 23:10:13.479873 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.479752    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" event={"ID":"3a94edad-54be-4e70-bfcb-8ba8c2b6374c","Type":"ContainerDied","Data":"4a9de364354d66c7710414134167b2bfda9c1cbbdd0e4f0e39f750c8e7cc10fe"}
Nov 01 23:10:13.529261 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.529232    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 23:10:13.548493 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.548470    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 23:10:13.643957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.643932    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644000    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644047    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644067    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644099    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644150    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644182    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644218    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644259    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644288    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644304    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644318    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644354    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644351    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644359    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644384    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644459    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.644538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644493    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo\") pod \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\" (UID: \"80b7dd35-ea1b-4285-88cf-fc5b404039f3\") "
Nov 01 23:10:13.645136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644546    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.645136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644588    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket\") pod \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\" (UID: \"3a94edad-54be-4e70-bfcb-8ba8c2b6374c\") "
Nov 01 23:10:13.645136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644599    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.645136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644641    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.645136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644854    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.645136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.644876    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645212    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645235    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645251    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645266    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645280    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645295    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645311    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645317    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.645469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645452    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.646016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.645754    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:10:13.646867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.646842    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "kube-api-access-9d62p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:13.647001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.646925    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "kube-api-access-4rm4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:13.647088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.647064    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.647248 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.647226    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:13.647346 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.647241    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:10:13.647469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.647450    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:10:13.647962 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.647947    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token" pod "3a94edad-54be-4e70-bfcb-8ba8c2b6374c" (UID: "3a94edad-54be-4e70-bfcb-8ba8c2b6374c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:13.648334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.648320    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo" pod "80b7dd35-ea1b-4285-88cf-fc5b404039f3" (UID: "80b7dd35-ea1b-4285-88cf-fc5b404039f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746788    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d62p\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-kube-api-access-9d62p\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746812    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746823    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746831    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746841    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80b7dd35-ea1b-4285-88cf-fc5b404039f3-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746850    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rm4n\" (UniqueName: \"kubernetes.io/projected/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-kube-api-access-4rm4n\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.746856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746858    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.747199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746867    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.747199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746876    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.747199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746884    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80b7dd35-ea1b-4285-88cf-fc5b404039f3-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:13.747199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:13.746893    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3a94edad-54be-4e70-bfcb-8ba8c2b6374c-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:14.493302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.493217    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"
Nov 01 23:10:14.493302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.493242    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs" event={"ID":"80b7dd35-ea1b-4285-88cf-fc5b404039f3","Type":"ContainerDied","Data":"fb4e3fb4df07ca650823a37c8d0744f08f0401643fc3da2507a658e946c174b8"}
Nov 01 23:10:14.493595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.493322    4531 scope.go:122] "RemoveContainer" containerID="7b0d6dd705674572a99557d4d6679b1610110ba2a328da4926520f1500257b4b"
Nov 01 23:10:14.495395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.495367    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d" event={"ID":"3a94edad-54be-4e70-bfcb-8ba8c2b6374c","Type":"ContainerDied","Data":"833f0f95a66f108ca00a1dc6df38d57f0d19291c462ffa281b7ae641ff549515"}
Nov 01 23:10:14.495505 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.495471    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"
Nov 01 23:10:14.506646 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.506626    4531 scope.go:122] "RemoveContainer" containerID="4a9de364354d66c7710414134167b2bfda9c1cbbdd0e4f0e39f750c8e7cc10fe"
Nov 01 23:10:14.650905 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.650877    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"]
Nov 01 23:10:14.697617 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.697582    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-tvx5d"]
Nov 01 23:10:14.799830 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.799664    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"]
Nov 01 23:10:14.805188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.805162    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-sl6fs"]
Nov 01 23:10:14.817769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.817744    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" path="/var/lib/kubelet/pods/3a94edad-54be-4e70-bfcb-8ba8c2b6374c/volumes"
Nov 01 23:10:14.818272 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:14.818256    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" path="/var/lib/kubelet/pods/80b7dd35-ea1b-4285-88cf-fc5b404039f3/volumes"
Nov 01 23:10:15.475571 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:15.475537    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:16.582870 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:16.582842    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:20.313740 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:20.313695    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:25.056682 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:25.056650    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:29.839995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:29.839961    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:31.443363 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443327    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"]
Nov 01 23:10:31.443893 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443879    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ea78909a-3497-4179-8f45-7c6221b15172" containerName="revoke-keys"
Nov 01 23:10:31.443951 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443893    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea78909a-3497-4179-8f45-7c6221b15172" containerName="revoke-keys"
Nov 01 23:10:31.443951 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443920    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1bc54405-3858-4828-8b79-8c0b35bc40f8" containerName="main"
Nov 01 23:10:31.443951 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443926    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bc54405-3858-4828-8b79-8c0b35bc40f8" containerName="main"
Nov 01 23:10:31.443951 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443942    4531 cpu_manager.go:414] "containerMap: removing container" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy"
Nov 01 23:10:31.443951 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443948    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443959    4531 cpu_manager.go:414] "containerMap: removing container" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443964    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443974    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.443978    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.444060    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a94edad-54be-4e70-bfcb-8ba8c2b6374c" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.444071    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea78909a-3497-4179-8f45-7c6221b15172" containerName="revoke-keys"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.444081    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bc54405-3858-4828-8b79-8c0b35bc40f8" containerName="main"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.444089    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b7dd35-ea1b-4285-88cf-fc5b404039f3" containerName="istio-proxy"
Nov 01 23:10:31.444113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.444096    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="72da0505-3e3f-4764-a695-faa46ae5a8d3" containerName="istio-proxy"
Nov 01 23:10:31.447842 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.447828    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.500866 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.500836    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:10:31.533739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.533696    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.533933 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.533857    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrg9s\" (UniqueName: \"kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.534008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.533966    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.546882 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.546860    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:10:31.550281 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.550248    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"]
Nov 01 23:10:31.634997 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.634961    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrg9s\" (UniqueName: \"kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.635211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.635054    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.635211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.635105    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.635742 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.635707    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.637386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.637363    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.689263 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.689236    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrg9s\" (UniqueName: \"kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s\") pod \"maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:31.769275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:31.769246    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:32.144434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:32.144366    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"]
Nov 01 23:10:32.148848 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:10:32.148823    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9769c3e8_61e2_46ec_bff3_fc29ef3bac1e.slice/crio-8e83a8c7631c7fefd6d015412c67b4e44a6aaa0cbbed05340954bf43da80a4a7 WatchSource:0}: Error finding container 8e83a8c7631c7fefd6d015412c67b4e44a6aaa0cbbed05340954bf43da80a4a7: Status 404 returned error can't find the container with id 8e83a8c7631c7fefd6d015412c67b4e44a6aaa0cbbed05340954bf43da80a4a7
Nov 01 23:10:32.633347 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:32.633315    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx" event={"ID":"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e","Type":"ContainerStarted","Data":"57ca563580bd99a3d8bb9cc99c157f0763cb6441a8fedf7865a57cbc0906cf4a"}
Nov 01 23:10:32.633697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:32.633371    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx" event={"ID":"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e","Type":"ContainerStarted","Data":"8e83a8c7631c7fefd6d015412c67b4e44a6aaa0cbbed05340954bf43da80a4a7"}
Nov 01 23:10:32.743229 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:32.743147    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx" podStartSLOduration=1.743133775 podStartE2EDuration="1.743133775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:32.739877695 +0000 UTC m=+2814.057527695" watchObservedRunningTime="2026-11-01 23:10:32.743133775 +0000 UTC m=+2814.060783777"
Nov 01 23:10:33.642673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:33.642639    4531 generic.go:356] "Generic (PLEG): container finished" podID="9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" containerID="57ca563580bd99a3d8bb9cc99c157f0763cb6441a8fedf7865a57cbc0906cf4a" exitCode=0
Nov 01 23:10:33.643088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:33.642719    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx" event={"ID":"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e","Type":"ContainerDied","Data":"57ca563580bd99a3d8bb9cc99c157f0763cb6441a8fedf7865a57cbc0906cf4a"}
Nov 01 23:10:34.340065 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.340025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:34.801273 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.801249    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:34.864541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.864516    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca\") pod \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") "
Nov 01 23:10:34.864712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.864580    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s\" (UniqueName: \"kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s\") pod \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") "
Nov 01 23:10:34.864822 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.864708    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca\") pod \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\" (UID: \"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e\") "
Nov 01 23:10:34.865265 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.865085    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca" pod "9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" (UID: "9769c3e8-61e2-46ec-bff3-fc29ef3bac1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:10:34.865466 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.865432    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:34.866834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.866801    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca" pod "9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" (UID: "9769c3e8-61e2-46ec-bff3-fc29ef3bac1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:10:34.866920 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.866907    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s" pod "9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" (UID: "9769c3e8-61e2-46ec-bff3-fc29ef3bac1e"). InnerVolumeSpecName "kube-api-access-vrg9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:34.966694 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.966666    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrg9s\" (UniqueName: \"kubernetes.io/projected/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-kube-api-access-vrg9s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:34.966866 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:34.966696    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:35.058306 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:35.058269    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:35.658080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:35.658050    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx" event={"ID":"9769c3e8-61e2-46ec-bff3-fc29ef3bac1e","Type":"ContainerDied","Data":"8e83a8c7631c7fefd6d015412c67b4e44a6aaa0cbbed05340954bf43da80a4a7"}
Nov 01 23:10:35.658080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:35.658079    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"
Nov 01 23:10:35.658305 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:35.658095    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e83a8c7631c7fefd6d015412c67b4e44a6aaa0cbbed05340954bf43da80a4a7"
Nov 01 23:10:39.990225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:39.990172    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:42.079585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:42.079555    4531 scope.go:122] "RemoveContainer" containerID="a44d4748839a9351ecc28ad99c2d4784122da861c15c9e2ebe91094d589d2e74"
Nov 01 23:10:42.092171 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:42.092152    4531 scope.go:122] "RemoveContainer" containerID="c920667b32916fbaaec9e30fee928801fb418e8cd610786f2d44c4b83b0c158e"
Nov 01 23:10:42.526399 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:10:42.526356    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:10:43.486906 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:43.486867    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:43.730751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:43.730697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:43.950620 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:43.950587    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:44.156585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:44.156549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:45.930015 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:45.929976    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:47.854118 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:47.854086    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:48.192195 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:48.192110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:48.427490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:48.427458    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"]
Nov 01 23:10:48.473492 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:48.473459    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-675d1d-84dwx"]
Nov 01 23:10:48.817586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:48.817511    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" path="/var/lib/kubelet/pods/9769c3e8-61e2-46ec-bff3-fc29ef3bac1e/volumes"
Nov 01 23:10:48.890924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:48.890881    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:50.942419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:50.942192    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:52.995038 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:52.995003    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"]
Nov 01 23:10:52.995625 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:52.995610    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" containerName="revoke-keys"
Nov 01 23:10:52.995625 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:52.995625    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" containerName="revoke-keys"
Nov 01 23:10:52.995766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:52.995753    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9769c3e8-61e2-46ec-bff3-fc29ef3bac1e" containerName="revoke-keys"
Nov 01 23:10:52.999302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:52.999287    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.049661 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.049616    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:10:53.049874 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.049859    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:10:53.099478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.099454    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"]
Nov 01 23:10:53.161613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.161586    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.161771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.161634    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd86f\" (UniqueName: \"kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.161771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.161759    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.262892 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.262835    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.262892 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.262885    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd86f\" (UniqueName: \"kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.263113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.262930    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.263529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.263511    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.265123 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.265103    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.316851 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.316828    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd86f\" (UniqueName: \"kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f\") pod \"maas-api-revoke-sub-89bf61fd-xs4pq\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.612642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.612571    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:53.989630 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:53.989605    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"]
Nov 01 23:10:53.992431 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:10:53.992402    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e40b2f7_1082_453b_b708_8aa401beb6b2.slice/crio-aa8902728ab17400387f53d96bda77e34b320c0c8955ee0ebc99f44c525ce359 WatchSource:0}: Error finding container aa8902728ab17400387f53d96bda77e34b320c0c8955ee0ebc99f44c525ce359: Status 404 returned error can't find the container with id aa8902728ab17400387f53d96bda77e34b320c0c8955ee0ebc99f44c525ce359
Nov 01 23:10:54.792928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:54.792890    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq" event={"ID":"1e40b2f7-1082-453b-b708-8aa401beb6b2","Type":"ContainerStarted","Data":"1d63789fe91f5b5f1cad7e4ecc098a41e7e99c809ace85c45c1190b38e3ce410"}
Nov 01 23:10:54.793355 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:54.792955    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq" event={"ID":"1e40b2f7-1082-453b-b708-8aa401beb6b2","Type":"ContainerStarted","Data":"aa8902728ab17400387f53d96bda77e34b320c0c8955ee0ebc99f44c525ce359"}
Nov 01 23:10:54.902776 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:54.902714    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq" podStartSLOduration=2.9027013200000003 podStartE2EDuration="2.90270132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:54.900445469 +0000 UTC m=+2836.218095470" watchObservedRunningTime="2026-11-01 23:10:54.90270132 +0000 UTC m=+2836.220351321"
Nov 01 23:10:55.509867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:55.509837    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:55.705917 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:55.705885    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:55.803653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:55.803564    4531 generic.go:356] "Generic (PLEG): container finished" podID="1e40b2f7-1082-453b-b708-8aa401beb6b2" containerID="1d63789fe91f5b5f1cad7e4ecc098a41e7e99c809ace85c45c1190b38e3ce410" exitCode=0
Nov 01 23:10:55.803653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:55.803637    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq" event={"ID":"1e40b2f7-1082-453b-b708-8aa401beb6b2","Type":"ContainerDied","Data":"1d63789fe91f5b5f1cad7e4ecc098a41e7e99c809ace85c45c1190b38e3ce410"}
Nov 01 23:10:55.896059 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:55.896030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:56.096341 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.096265    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:10:56.578377 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.578343    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"]
Nov 01 23:10:56.582547 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.582531    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.638001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.637971    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"]
Nov 01 23:10:56.702812 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.702773    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.702957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.702824    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.703008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.702982    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snbx4\" (UniqueName: \"kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.804599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.804561    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snbx4\" (UniqueName: \"kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.805076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.804668    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.805076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.804695    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.805470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.805451    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.807125 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.807097    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.858172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.858096    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snbx4\" (UniqueName: \"kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4\") pod \"maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.897075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.896910    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:10:56.976009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:56.975981    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:57.110776 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.110697    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca\") pod \"1e40b2f7-1082-453b-b708-8aa401beb6b2\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") "
Nov 01 23:10:57.110930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.110790    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca\") pod \"1e40b2f7-1082-453b-b708-8aa401beb6b2\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") "
Nov 01 23:10:57.111049 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.111029    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f\" (UniqueName: \"kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f\") pod \"1e40b2f7-1082-453b-b708-8aa401beb6b2\" (UID: \"1e40b2f7-1082-453b-b708-8aa401beb6b2\") "
Nov 01 23:10:57.111200 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.111174    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca" pod "1e40b2f7-1082-453b-b708-8aa401beb6b2" (UID: "1e40b2f7-1082-453b-b708-8aa401beb6b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:10:57.111508 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.111486    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:57.112905 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.112880    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca" pod "1e40b2f7-1082-453b-b708-8aa401beb6b2" (UID: "1e40b2f7-1082-453b-b708-8aa401beb6b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:10:57.113101 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.113081    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f" pod "1e40b2f7-1082-453b-b708-8aa401beb6b2" (UID: "1e40b2f7-1082-453b-b708-8aa401beb6b2"). InnerVolumeSpecName "kube-api-access-xd86f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:10:57.212792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.212763    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd86f\" (UniqueName: \"kubernetes.io/projected/1e40b2f7-1082-453b-b708-8aa401beb6b2-kube-api-access-xd86f\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:57.212792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.212790    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e40b2f7-1082-453b-b708-8aa401beb6b2-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:10:57.284025 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.283997    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"]
Nov 01 23:10:57.287613 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:10:57.287588    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd91c5a_ab5f_45fc_a4d2_72c9f9b75618.slice/crio-2be408f7c09f2b2c438d51bb23890f6c4799ae923cf7125ce2d642ba9a621e8a WatchSource:0}: Error finding container 2be408f7c09f2b2c438d51bb23890f6c4799ae923cf7125ce2d642ba9a621e8a: Status 404 returned error can't find the container with id 2be408f7c09f2b2c438d51bb23890f6c4799ae923cf7125ce2d642ba9a621e8a
Nov 01 23:10:57.832036 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.831993    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9" event={"ID":"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618","Type":"ContainerStarted","Data":"af0c4e067e6d5a30750dbacfc5595c71cbcd931b53d1ad206f7c5b6516bf363a"}
Nov 01 23:10:57.832456 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.832091    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9" event={"ID":"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618","Type":"ContainerStarted","Data":"2be408f7c09f2b2c438d51bb23890f6c4799ae923cf7125ce2d642ba9a621e8a"}
Nov 01 23:10:57.833669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.833644    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"
Nov 01 23:10:57.833669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.833654    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq" event={"ID":"1e40b2f7-1082-453b-b708-8aa401beb6b2","Type":"ContainerDied","Data":"aa8902728ab17400387f53d96bda77e34b320c0c8955ee0ebc99f44c525ce359"}
Nov 01 23:10:57.833863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.833692    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa8902728ab17400387f53d96bda77e34b320c0c8955ee0ebc99f44c525ce359"
Nov 01 23:10:57.942452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:57.942400    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9" podStartSLOduration=1.94238474 podStartE2EDuration="1.94238474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:10:57.937271 +0000 UTC m=+2839.254921001" watchObservedRunningTime="2026-11-01 23:10:57.94238474 +0000 UTC m=+2839.260034742"
Nov 01 23:10:58.841754 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:58.841692    4531 generic.go:356] "Generic (PLEG): container finished" podID="8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" containerID="af0c4e067e6d5a30750dbacfc5595c71cbcd931b53d1ad206f7c5b6516bf363a" exitCode=0
Nov 01 23:10:58.842153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:58.841772    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9" event={"ID":"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618","Type":"ContainerDied","Data":"af0c4e067e6d5a30750dbacfc5595c71cbcd931b53d1ad206f7c5b6516bf363a"}
Nov 01 23:10:59.560657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:10:59.560623    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:00.015579 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.015550    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:11:00.146105 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.146027    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca\") pod \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") "
Nov 01 23:11:00.146255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.146118    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca\") pod \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") "
Nov 01 23:11:00.146255 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.146239    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4\" (UniqueName: \"kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4\") pod \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\" (UID: \"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618\") "
Nov 01 23:11:00.146435 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.146409    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca" pod "8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" (UID: "8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:00.147292 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.147266    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.148180 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.148156    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca" pod "8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" (UID: "8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:00.148383 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.148359    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4" pod "8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" (UID: "8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618"). InnerVolumeSpecName "kube-api-access-snbx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:00.248315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.248283    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.248315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.248311    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snbx4\" (UniqueName: \"kubernetes.io/projected/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618-kube-api-access-snbx4\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:00.468900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.468867    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:00.870046 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.870001    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9" event={"ID":"8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618","Type":"ContainerDied","Data":"2be408f7c09f2b2c438d51bb23890f6c4799ae923cf7125ce2d642ba9a621e8a"}
Nov 01 23:11:00.870253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.870073    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be408f7c09f2b2c438d51bb23890f6c4799ae923cf7125ce2d642ba9a621e8a"
Nov 01 23:11:00.870253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:00.870138    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"
Nov 01 23:11:02.101104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.101068    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:02.300106 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.300064    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:02.490738 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.490665    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:02.536121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.536081    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"]
Nov 01 23:11:02.539211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.539189    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3eb631-dn2z6"]
Nov 01 23:11:02.817668 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.817586    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d92bd53-1bec-498f-b5c2-55e5868f9a75" path="/var/lib/kubelet/pods/2d92bd53-1bec-498f-b5c2-55e5868f9a75/volumes"
Nov 01 23:11:02.986577 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:02.986550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:07.440532 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:07.440496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:07.709507 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:07.709474    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:09.020085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.020034    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:09.213640 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.213605    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:09.367205 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.366121    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"]
Nov 01 23:11:09.412923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.412874    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:09.481709 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.481675    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"]
Nov 01 23:11:09.793701 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.793624    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:09.983351 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:09.983316    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:10.297558 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.297521    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"]
Nov 01 23:11:10.298328 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.298305    4531 cpu_manager.go:414] "containerMap: removing container" podUID="8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" containerName="revoke-keys"
Nov 01 23:11:10.298442 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.298331    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" containerName="revoke-keys"
Nov 01 23:11:10.298442 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.298374    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1e40b2f7-1082-453b-b708-8aa401beb6b2" containerName="revoke-keys"
Nov 01 23:11:10.298442 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.298382    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e40b2f7-1082-453b-b708-8aa401beb6b2" containerName="revoke-keys"
Nov 01 23:11:10.298598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.298541    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e40b2f7-1082-453b-b708-8aa401beb6b2" containerName="revoke-keys"
Nov 01 23:11:10.298598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.298566    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" containerName="revoke-keys"
Nov 01 23:11:10.302066 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.302046    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.352283 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.352255    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:11:10.352631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.352597    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:11:10.358785 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.358759    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"]
Nov 01 23:11:10.359468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.359436    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.359558 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.359499    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l8v4\" (UniqueName: \"kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.359635 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.359619    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.461113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.461087    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.461286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.461197    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.461286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.461222    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l8v4\" (UniqueName: \"kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.461942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.461912    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.463645 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.463616    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.516562 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.516531    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l8v4\" (UniqueName: \"kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4\") pod \"maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:10.618657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:10.618591    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:11.003032 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:11.002987    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"]
Nov 01 23:11:11.007221 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:11:11.007190    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b7a2f0c_658f_4f3f_9eb3_37c7d96d9b87.slice/crio-4f5a5ff7266f22eb2fa6feac7eed08017c48592e05cc2bb7429856fea3a2a90c WatchSource:0}: Error finding container 4f5a5ff7266f22eb2fa6feac7eed08017c48592e05cc2bb7429856fea3a2a90c: Status 404 returned error can't find the container with id 4f5a5ff7266f22eb2fa6feac7eed08017c48592e05cc2bb7429856fea3a2a90c
Nov 01 23:11:11.953465 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:11.953429    4531 generic.go:356] "Generic (PLEG): container finished" podID="7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" containerID="7dacdaf4c31fe5344b13bb1a279d9987ae3a9ba1fac3d8f429c8d39d6dc3fef0" exitCode=0
Nov 01 23:11:11.953931 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:11.953522    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj" event={"ID":"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87","Type":"ContainerDied","Data":"7dacdaf4c31fe5344b13bb1a279d9987ae3a9ba1fac3d8f429c8d39d6dc3fef0"}
Nov 01 23:11:11.953931 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:11.953599    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj" event={"ID":"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87","Type":"ContainerStarted","Data":"4f5a5ff7266f22eb2fa6feac7eed08017c48592e05cc2bb7429856fea3a2a90c"}
Nov 01 23:11:12.499913 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:12.499877    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:12.528900 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:11:12.528866    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:11:12.743828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:12.743795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:13.109953 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.109927    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:13.195502 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.195479    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4\" (UniqueName: \"kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4\") pod \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") "
Nov 01 23:11:13.195651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.195594    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca\") pod \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") "
Nov 01 23:11:13.195651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.195622    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca\") pod \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\" (UID: \"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87\") "
Nov 01 23:11:13.195989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.195963    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca" pod "7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" (UID: "7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:13.197524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.197500    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4" pod "7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" (UID: "7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87"). InnerVolumeSpecName "kube-api-access-9l8v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:13.197585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.197563    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca" pod "7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" (UID: "7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:13.296861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.296801    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:13.296861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.296826    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:13.296861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.296836    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l8v4\" (UniqueName: \"kubernetes.io/projected/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87-kube-api-access-9l8v4\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:13.783010 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.782960    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"]
Nov 01 23:11:13.783538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.783523    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" containerName="revoke-keys"
Nov 01 23:11:13.783619 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.783538    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" containerName="revoke-keys"
Nov 01 23:11:13.783672 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.783639    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" containerName="revoke-keys"
Nov 01 23:11:13.787616 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.787592    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:13.842628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.842599    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"]
Nov 01 23:11:13.903314 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.903278    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:13.903475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.903422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2tnk\" (UniqueName: \"kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:13.903566 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.903500    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:13.970670 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.970635    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj" event={"ID":"7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87","Type":"ContainerDied","Data":"4f5a5ff7266f22eb2fa6feac7eed08017c48592e05cc2bb7429856fea3a2a90c"}
Nov 01 23:11:13.970670 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.970656    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"
Nov 01 23:11:13.970909 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:13.970693    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f5a5ff7266f22eb2fa6feac7eed08017c48592e05cc2bb7429856fea3a2a90c"
Nov 01 23:11:14.004559 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.004531    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.004679 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.004630    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.004719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.004687    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2tnk\" (UniqueName: \"kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.005324 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.005306    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.006934 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.006914    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.058712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.058666    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2tnk\" (UniqueName: \"kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk\") pod \"maas-api-revoke-sub-ad6086ab-75v8v\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.102691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.102666    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:14.366797 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.366692    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr" podUID="9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" containerName="payload-processing" containerID="cri-o://90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61" gracePeriod=25
Nov 01 23:11:14.479375 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.479346    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"]
Nov 01 23:11:14.482751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.482683    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerName="payload-pre-processing" containerID="cri-o://1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98" gracePeriod=25
Nov 01 23:11:14.483368 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:11:14.483079    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89175acc_4741_45d4_976e_d018f1f09968.slice/crio-c9ee629f5c555bdb83b537ce6ea52fe06f3219ca1b3f396a9e47e3ef9f0060d5 WatchSource:0}: Error finding container c9ee629f5c555bdb83b537ce6ea52fe06f3219ca1b3f396a9e47e3ef9f0060d5: Status 404 returned error can't find the container with id c9ee629f5c555bdb83b537ce6ea52fe06f3219ca1b3f396a9e47e3ef9f0060d5
Nov 01 23:11:14.625374 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.625318    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:11:14.713507 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.713480    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv\" (UniqueName: \"kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv\") pod \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") "
Nov 01 23:11:14.713648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.713593    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume\") pod \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\" (UID: \"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3\") "
Nov 01 23:11:14.714103 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.714060    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume" pod "9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" (UID: "9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:14.715841 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.715819    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv" pod "9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" (UID: "9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3"). InnerVolumeSpecName "kube-api-access-qrvnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:14.718949 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.718927    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"]
Nov 01 23:11:14.814486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.814461    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvnv\" (UniqueName: \"kubernetes.io/projected/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-kube-api-access-qrvnv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.814628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.814488    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:14.878511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.878433    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:14.952544 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.952515    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:14.987362 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.987307    4531 generic.go:356] "Generic (PLEG): container finished" podID="9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" containerID="90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61" exitCode=0
Nov 01 23:11:14.987610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.987578    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr" event={"ID":"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3","Type":"ContainerDied","Data":"90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61"}
Nov 01 23:11:14.987773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.987686    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr" event={"ID":"9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3","Type":"ContainerDied","Data":"1efb4c47c2891c3b4b9de4e21cd977b5140e327aaa4a9b21db97a55dfd383465"}
Nov 01 23:11:14.987773 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.987754    4531 scope.go:122] "RemoveContainer" containerID="90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61"
Nov 01 23:11:14.987972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.987954    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"
Nov 01 23:11:14.990859 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.990815    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v" event={"ID":"89175acc-4741-45d4-976e-d018f1f09968","Type":"ContainerStarted","Data":"ff6e9449d43ad4d775d453af03f6dfb2658f1749a970ab925decb400d08b59aa"}
Nov 01 23:11:14.990988 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.990887    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v" event={"ID":"89175acc-4741-45d4-976e-d018f1f09968","Type":"ContainerStarted","Data":"c9ee629f5c555bdb83b537ce6ea52fe06f3219ca1b3f396a9e47e3ef9f0060d5"}
Nov 01 23:11:14.999403 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.999381    4531 scope.go:122] "RemoveContainer" containerID="90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61"
Nov 01 23:11:14.999680 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:11:14.999658    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61\": container with ID starting with 90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61 not found: ID does not exist" containerID="90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61"
Nov 01 23:11:14.999775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:14.999687    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61"} err="failed to get container status \"90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61\": rpc error: code = NotFound desc = could not find container \"90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61\": container with ID starting with 90f953fb7482c4c0d91436da7d87a01f22d56c1732163cae580163d501517c61 not found: ID does not exist"
Nov 01 23:11:15.146265 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:15.146193    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"]
Nov 01 23:11:15.149576 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:15.149553    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4af44d-7bfdc584bf-wlscr"]
Nov 01 23:11:15.254341 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:15.254293    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v" podStartSLOduration=2.254282184 podStartE2EDuration="2.254282184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:15.252027349 +0000 UTC m=+2856.569677350" watchObservedRunningTime="2026-11-01 23:11:15.254282184 +0000 UTC m=+2856.571932200"
Nov 01 23:11:16.001015 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:16.000981    4531 generic.go:356] "Generic (PLEG): container finished" podID="89175acc-4741-45d4-976e-d018f1f09968" containerID="ff6e9449d43ad4d775d453af03f6dfb2658f1749a970ab925decb400d08b59aa" exitCode=0
Nov 01 23:11:16.001405 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:16.001014    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v" event={"ID":"89175acc-4741-45d4-976e-d018f1f09968","Type":"ContainerDied","Data":"ff6e9449d43ad4d775d453af03f6dfb2658f1749a970ab925decb400d08b59aa"}
Nov 01 23:11:16.817575 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:16.817543    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" path="/var/lib/kubelet/pods/9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3/volumes"
Nov 01 23:11:17.142232 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.142207    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:17.236534 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.236503    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk\" (UniqueName: \"kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk\") pod \"89175acc-4741-45d4-976e-d018f1f09968\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") "
Nov 01 23:11:17.236710 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.236691    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca\") pod \"89175acc-4741-45d4-976e-d018f1f09968\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") "
Nov 01 23:11:17.236788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.236776    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca\") pod \"89175acc-4741-45d4-976e-d018f1f09968\" (UID: \"89175acc-4741-45d4-976e-d018f1f09968\") "
Nov 01 23:11:17.237107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.237084    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca" pod "89175acc-4741-45d4-976e-d018f1f09968" (UID: "89175acc-4741-45d4-976e-d018f1f09968"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:17.237228 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.237210    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89175acc-4741-45d4-976e-d018f1f09968-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:17.238705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.238679    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk" pod "89175acc-4741-45d4-976e-d018f1f09968" (UID: "89175acc-4741-45d4-976e-d018f1f09968"). InnerVolumeSpecName "kube-api-access-x2tnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:17.238705 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.238696    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca" pod "89175acc-4741-45d4-976e-d018f1f09968" (UID: "89175acc-4741-45d4-976e-d018f1f09968"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:17.338772 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.338719    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89175acc-4741-45d4-976e-d018f1f09968-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:17.338919 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.338775    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2tnk\" (UniqueName: \"kubernetes.io/projected/89175acc-4741-45d4-976e-d018f1f09968-kube-api-access-x2tnk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:17.442437 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:17.442351    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:18.015662 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:18.015631    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"
Nov 01 23:11:18.015861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:18.015665    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v" event={"ID":"89175acc-4741-45d4-976e-d018f1f09968","Type":"ContainerDied","Data":"c9ee629f5c555bdb83b537ce6ea52fe06f3219ca1b3f396a9e47e3ef9f0060d5"}
Nov 01 23:11:18.015861 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:18.015709    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9ee629f5c555bdb83b537ce6ea52fe06f3219ca1b3f396a9e47e3ef9f0060d5"
Nov 01 23:11:21.892148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:21.892112    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"]
Nov 01 23:11:22.202359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:22.202324    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:22.382818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:22.382783    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:24.952418 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:24.952381    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:26.892753 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:26.892702    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerName="payload-pre-processing" containerID="cri-o://fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c" gracePeriod=25
Nov 01 23:11:27.124263 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:27.124225    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:29.739716 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:29.739670    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" containerID="cri-o://215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0" gracePeriod=45
Nov 01 23:11:29.814985 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:29.814941    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main" probeResult="failure" output="Get \"https://10.133.0.97:8000/ready\": dial tcp 10.133.0.97:8000: connect: connection refused"
Nov 01 23:11:30.009279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.009252    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 23:11:30.108796 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.108764    4531 generic.go:356] "Generic (PLEG): container finished" podID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerID="215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0" exitCode=0
Nov 01 23:11:30.108952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.108799    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"
Nov 01 23:11:30.108952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.108846    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" event={"ID":"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db","Type":"ContainerDied","Data":"215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0"}
Nov 01 23:11:30.108952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.108901    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h" event={"ID":"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db","Type":"ContainerDied","Data":"6f8dbfcb1774669b96592fccf5151350d641a6f31d4b90575eea13b959e91560"}
Nov 01 23:11:30.108952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.108923    4531 scope.go:122] "RemoveContainer" containerID="215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0"
Nov 01 23:11:30.121791 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.121521    4531 scope.go:122] "RemoveContainer" containerID="215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0"
Nov 01 23:11:30.121903 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:11:30.121879    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0\": container with ID starting with 215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0 not found: ID does not exist" containerID="215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0"
Nov 01 23:11:30.121957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.121914    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0"} err="failed to get container status \"215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0\": rpc error: code = NotFound desc = could not find container \"215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0\": container with ID starting with 215d131aecd9ac7a3cd2adcf48bf7421ced9548ca89c67a18c76b4d64c8866b0 not found: ID does not exist"
Nov 01 23:11:30.185901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.185870    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm\") pod \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") "
Nov 01 23:11:30.186102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.185968    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache\") pod \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") "
Nov 01 23:11:30.186102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.186024    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs\" (UniqueName: \"kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs\") pod \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") "
Nov 01 23:11:30.186102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.186058    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home\") pod \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") "
Nov 01 23:11:30.186295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.186132    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir\") pod \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") "
Nov 01 23:11:30.186295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.186177    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw\" (UniqueName: \"kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw\") pod \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\" (UID: \"75ef3a0f-3447-4ed5-85d0-b36c86bbe5db\") "
Nov 01 23:11:30.187378 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.187220    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache" pod "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" (UID: "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.187684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.187565    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home" pod "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" (UID: "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.187684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.187643    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir" pod "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" (UID: "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.190293 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.189547    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw" pod "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" (UID: "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db"). InnerVolumeSpecName "kube-api-access-k5snw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:30.190293 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.190082    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs" pod "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" (UID: "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:30.190628 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.190605    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm" pod "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" (UID: "75ef3a0f-3447-4ed5-85d0-b36c86bbe5db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:11:30.287287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.287212    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5snw\" (UniqueName: \"kubernetes.io/projected/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-kube-api-access-k5snw\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.287287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.287243    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.287287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.287257    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.287287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.287271    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.287287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.287283    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.287669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.287296    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:30.524026 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.523979    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"]
Nov 01 23:11:30.570420 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.570346    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-a6dd2f/distinct-w6-a6dd2f-kserve-54f69f56bd-w848h"]
Nov 01 23:11:30.816132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:30.816090    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" path="/var/lib/kubelet/pods/75ef3a0f-3447-4ed5-85d0-b36c86bbe5db/volumes"
Nov 01 23:11:31.933745 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:31.933700    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:34.375997 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.375955    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:34.750360 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.750337    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:11:34.835991 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.835966    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume\") pod \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") "
Nov 01 23:11:34.836137 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.836033    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c\" (UniqueName: \"kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c\") pod \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\" (UID: \"d454cd9d-fea6-400b-b53e-f6ecc01d0619\") "
Nov 01 23:11:34.836319 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.836294    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume" pod "d454cd9d-fea6-400b-b53e-f6ecc01d0619" (UID: "d454cd9d-fea6-400b-b53e-f6ecc01d0619"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:34.838037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.838015    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c" pod "d454cd9d-fea6-400b-b53e-f6ecc01d0619" (UID: "d454cd9d-fea6-400b-b53e-f6ecc01d0619"). InnerVolumeSpecName "kube-api-access-zgd9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:34.937009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.936949    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgd9c\" (UniqueName: \"kubernetes.io/projected/d454cd9d-fea6-400b-b53e-f6ecc01d0619-kube-api-access-zgd9c\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:34.937009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:34.936973    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d454cd9d-fea6-400b-b53e-f6ecc01d0619-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:35.144567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.144532    4531 generic.go:356] "Generic (PLEG): container finished" podID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerID="1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98" exitCode=0
Nov 01 23:11:35.144708 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.144580    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"
Nov 01 23:11:35.144708 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.144611    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" event={"ID":"d454cd9d-fea6-400b-b53e-f6ecc01d0619","Type":"ContainerDied","Data":"1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98"}
Nov 01 23:11:35.144708 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.144668    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk" event={"ID":"d454cd9d-fea6-400b-b53e-f6ecc01d0619","Type":"ContainerDied","Data":"672fb9839a88d3d9505f3a816ccde9f177f46245985961c1935a5aeba766c420"}
Nov 01 23:11:35.144708 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.144691    4531 scope.go:122] "RemoveContainer" containerID="1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98"
Nov 01 23:11:35.155548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.155529    4531 scope.go:122] "RemoveContainer" containerID="1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98"
Nov 01 23:11:35.155828 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:11:35.155811    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98\": container with ID starting with 1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98 not found: ID does not exist" containerID="1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98"
Nov 01 23:11:35.155894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.155835    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98"} err="failed to get container status \"1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98\": rpc error: code = NotFound desc = could not find container \"1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98\": container with ID starting with 1c307231b7d859f59375428b4f12dc38cecf53d889b7f2a573373f9f8e680e98 not found: ID does not exist"
Nov 01 23:11:35.299742 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.299701    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"]
Nov 01 23:11:35.345787 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:35.345755    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4af44d-6cc9c948b7-lgjxk"]
Nov 01 23:11:36.760901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:36.760859    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:36.818137 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:36.818103    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" path="/var/lib/kubelet/pods/d454cd9d-fea6-400b-b53e-f6ecc01d0619/volumes"
Nov 01 23:11:41.479302 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:41.479255    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:42.144485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:42.144460    4531 scope.go:122] "RemoveContainer" containerID="604fbe3ab83dc6c7c79036ab061e37e064e2a7a2f04caf327cb0295c129f3d62"
Nov 01 23:11:42.157291 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:42.157264    4531 scope.go:122] "RemoveContainer" containerID="0e800ba8a5ad337541161bfba2eab5b9208c257b55aae45b1c14d3c8a5c86cfc"
Nov 01 23:11:42.526539 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:11:42.526504    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:11:44.100529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:44.100490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:44.375968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:44.375886    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:11:45.683484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:45.683448    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:47.157867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.157809    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 23:11:47.230244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.230212    4531 generic.go:356] "Generic (PLEG): container finished" podID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerID="fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c" exitCode=0
Nov 01 23:11:47.230419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.230264    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"
Nov 01 23:11:47.230419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.230293    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" event={"ID":"b902e77f-91d9-4f19-bfeb-f2fa1034b66f","Type":"ContainerDied","Data":"fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c"}
Nov 01 23:11:47.230419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.230358    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5" event={"ID":"b902e77f-91d9-4f19-bfeb-f2fa1034b66f","Type":"ContainerDied","Data":"f4b9c7b4dc22fc771c7028ec0149d72fcdcbbdc5a9ca7bab9e5ea7991800d3e3"}
Nov 01 23:11:47.230419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.230381    4531 scope.go:122] "RemoveContainer" containerID="fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c"
Nov 01 23:11:47.240956 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.240941    4531 scope.go:122] "RemoveContainer" containerID="fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c"
Nov 01 23:11:47.241222 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:11:47.241203    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c\": container with ID starting with fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c not found: ID does not exist" containerID="fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c"
Nov 01 23:11:47.241283 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.241231    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c"} err="failed to get container status \"fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c\": rpc error: code = NotFound desc = could not find container \"fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c\": container with ID starting with fe1a675bf16d80879b0deaa0fd6368c3a2eee827fa1aaef7668e6c690d9de24c not found: ID does not exist"
Nov 01 23:11:47.262660 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.262627    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd\" (UniqueName: \"kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd\") pod \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") "
Nov 01 23:11:47.262808 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.262677    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume\") pod \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\" (UID: \"b902e77f-91d9-4f19-bfeb-f2fa1034b66f\") "
Nov 01 23:11:47.263144 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.263113    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume" pod "b902e77f-91d9-4f19-bfeb-f2fa1034b66f" (UID: "b902e77f-91d9-4f19-bfeb-f2fa1034b66f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:47.263247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.263230    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:47.265317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.265289    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd" pod "b902e77f-91d9-4f19-bfeb-f2fa1034b66f" (UID: "b902e77f-91d9-4f19-bfeb-f2fa1034b66f"). InnerVolumeSpecName "kube-api-access-lknbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:47.313556 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.313523    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:47.363853 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.363823    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lknbd\" (UniqueName: \"kubernetes.io/projected/b902e77f-91d9-4f19-bfeb-f2fa1034b66f-kube-api-access-lknbd\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:47.516406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.516365    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:47.646978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.646939    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"]
Nov 01 23:11:47.653702 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.653679    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"]
Nov 01 23:11:47.693329 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.693304    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-a6dd2f-b75b6898c-52wz5"]
Nov 01 23:11:47.699599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:47.699575    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4af44d-szdx9"]
Nov 01 23:11:48.109644 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.109615    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:48.399488 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.399398    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"]
Nov 01 23:11:48.498058 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.498024    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:48.817541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.817511    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618" path="/var/lib/kubelet/pods/8bd91c5a-ab5f-45fc-a4d2-72c9f9b75618/volumes"
Nov 01 23:11:48.817912 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.817898    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" path="/var/lib/kubelet/pods/b902e77f-91d9-4f19-bfeb-f2fa1034b66f/volumes"
Nov 01 23:11:48.965522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.965488    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"]
Nov 01 23:11:48.966104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966084    4531 cpu_manager.go:414] "containerMap: removing container" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main"
Nov 01 23:11:48.966104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966104    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966118    4531 cpu_manager.go:414] "containerMap: removing container" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerName="payload-pre-processing"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966126    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerName="payload-pre-processing"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966159    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerName="payload-pre-processing"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966167    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerName="payload-pre-processing"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966188    4531 cpu_manager.go:414] "containerMap: removing container" podUID="89175acc-4741-45d4-976e-d018f1f09968" containerName="revoke-keys"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966195    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89175acc-4741-45d4-976e-d018f1f09968" containerName="revoke-keys"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966212    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" containerName="payload-processing"
Nov 01 23:11:48.966285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966218    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" containerName="payload-processing"
Nov 01 23:11:48.966706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966402    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ef3a0f-3447-4ed5-85d0-b36c86bbe5db" containerName="main"
Nov 01 23:11:48.966706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966419    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="d454cd9d-fea6-400b-b53e-f6ecc01d0619" containerName="payload-pre-processing"
Nov 01 23:11:48.966706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966434    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b902e77f-91d9-4f19-bfeb-f2fa1034b66f" containerName="payload-pre-processing"
Nov 01 23:11:48.966706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966445    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="89175acc-4741-45d4-976e-d018f1f09968" containerName="revoke-keys"
Nov 01 23:11:48.966706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.966454    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0d40d6-f0a9-43d3-9d01-7c5c36708ef3" containerName="payload-processing"
Nov 01 23:11:48.971584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:48.971560    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.022308 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.022286    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:11:49.027241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.027214    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"]
Nov 01 23:11:49.077073 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.076980    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:11:49.082829 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.082804    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.083027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.082861    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7l8h\" (UniqueName: \"kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.083027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.082986    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.184096 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.184065    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.184251 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.184115    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7l8h\" (UniqueName: \"kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.184251 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.184152    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.184804 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.184780    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.186448 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.186426    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.239055 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.239025    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7l8h\" (UniqueName: \"kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h\") pod \"maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.285571 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.285548    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:49.660892 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:49.660810    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"]
Nov 01 23:11:49.664553 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:11:49.664526    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f87df9_3e0b_4ceb_8de4_82ab72289f19.slice/crio-764f18d4202b22a09700163416d263aef34f76401807c8167e333b30b9e652ac WatchSource:0}: Error finding container 764f18d4202b22a09700163416d263aef34f76401807c8167e333b30b9e652ac: Status 404 returned error can't find the container with id 764f18d4202b22a09700163416d263aef34f76401807c8167e333b30b9e652ac
Nov 01 23:11:50.256657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:50.256583    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks" event={"ID":"b1f87df9-3e0b-4ceb-8de4-82ab72289f19","Type":"ContainerStarted","Data":"986330da053a4c0c6f23924ed6c0a66f289ec2330f9ea73b78cb244cba90756f"}
Nov 01 23:11:50.256657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:50.256638    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks" event={"ID":"b1f87df9-3e0b-4ceb-8de4-82ab72289f19","Type":"ContainerStarted","Data":"764f18d4202b22a09700163416d263aef34f76401807c8167e333b30b9e652ac"}
Nov 01 23:11:50.412046 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:50.411989    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks" podStartSLOduration=2.411974524 podStartE2EDuration="2.411974524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:50.362877615 +0000 UTC m=+2891.680527616" watchObservedRunningTime="2026-11-01 23:11:50.411974524 +0000 UTC m=+2891.729624525"
Nov 01 23:11:51.273604 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:51.273558    4531 generic.go:356] "Generic (PLEG): container finished" podID="b1f87df9-3e0b-4ceb-8de4-82ab72289f19" containerID="986330da053a4c0c6f23924ed6c0a66f289ec2330f9ea73b78cb244cba90756f" exitCode=0
Nov 01 23:11:51.274047 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:51.273649    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks" event={"ID":"b1f87df9-3e0b-4ceb-8de4-82ab72289f19","Type":"ContainerDied","Data":"986330da053a4c0c6f23924ed6c0a66f289ec2330f9ea73b78cb244cba90756f"}
Nov 01 23:11:52.407824 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.402189    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:52.537130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.537044    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:52.592550 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.592517    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:52.624157 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.624125    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca\") pod \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") "
Nov 01 23:11:52.624330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.624194    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h\" (UniqueName: \"kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h\") pod \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") "
Nov 01 23:11:52.624330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.624265    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca\") pod \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\" (UID: \"b1f87df9-3e0b-4ceb-8de4-82ab72289f19\") "
Nov 01 23:11:52.624518 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.624491    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca" pod "b1f87df9-3e0b-4ceb-8de4-82ab72289f19" (UID: "b1f87df9-3e0b-4ceb-8de4-82ab72289f19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:52.624664 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.624638    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:52.626395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.626368    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca" pod "b1f87df9-3e0b-4ceb-8de4-82ab72289f19" (UID: "b1f87df9-3e0b-4ceb-8de4-82ab72289f19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:52.626508 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.626490    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h" pod "b1f87df9-3e0b-4ceb-8de4-82ab72289f19" (UID: "b1f87df9-3e0b-4ceb-8de4-82ab72289f19"). InnerVolumeSpecName "kube-api-access-p7l8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:52.725243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.725208    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:52.725243 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.725237    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7l8h\" (UniqueName: \"kubernetes.io/projected/b1f87df9-3e0b-4ceb-8de4-82ab72289f19-kube-api-access-p7l8h\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:52.798177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:52.798105    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:53.293802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:53.293761    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks" event={"ID":"b1f87df9-3e0b-4ceb-8de4-82ab72289f19","Type":"ContainerDied","Data":"764f18d4202b22a09700163416d263aef34f76401807c8167e333b30b9e652ac"}
Nov 01 23:11:53.293802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:53.293794    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"
Nov 01 23:11:53.294021 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:53.293828    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764f18d4202b22a09700163416d263aef34f76401807c8167e333b30b9e652ac"
Nov 01 23:11:54.580606 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:54.580539    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:54.800972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:54.800937    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:55.045792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.045755    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:55.733474 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.733441    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"]
Nov 01 23:11:55.734023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.734007    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b1f87df9-3e0b-4ceb-8de4-82ab72289f19" containerName="revoke-keys"
Nov 01 23:11:55.734067 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.734023    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f87df9-3e0b-4ceb-8de4-82ab72289f19" containerName="revoke-keys"
Nov 01 23:11:55.734138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.734129    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f87df9-3e0b-4ceb-8de4-82ab72289f19" containerName="revoke-keys"
Nov 01 23:11:55.737699 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.737681    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.790263 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.790237    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"]
Nov 01 23:11:55.832086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.832056    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:11:55.832086 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.832064    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:11:55.859601 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.859576    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.859777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.859625    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hclc\" (UniqueName: \"kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.859777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.859644    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.961181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.961144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.961355 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.961212    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hclc\" (UniqueName: \"kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.961355 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.961245    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.961968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.961940    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:55.963825 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:55.963803    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:56.015687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:56.015623    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hclc\" (UniqueName: \"kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:56.051779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:56.051752    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:56.427318 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:56.427250    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"]
Nov 01 23:11:56.430459 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:11:56.430431    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82193017_a025_4ef2_9f35_460a50a4275c.slice/crio-77914ed5981810eddce622177466e6f761e6fdaddad4171b7be85e0956013695 WatchSource:0}: Error finding container 77914ed5981810eddce622177466e6f761e6fdaddad4171b7be85e0956013695: Status 404 returned error can't find the container with id 77914ed5981810eddce622177466e6f761e6fdaddad4171b7be85e0956013695
Nov 01 23:11:57.321972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:57.321926    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5" event={"ID":"82193017-a025-4ef2-9f35-460a50a4275c","Type":"ContainerStarted","Data":"3bb4d78a4716957eab983a3d7e73681742c24d46cb1e0494b49e49e6fd375bff"}
Nov 01 23:11:57.322360 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:57.322026    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5" event={"ID":"82193017-a025-4ef2-9f35-460a50a4275c","Type":"ContainerStarted","Data":"77914ed5981810eddce622177466e6f761e6fdaddad4171b7be85e0956013695"}
Nov 01 23:11:57.429450 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:57.429404    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5" podStartSLOduration=2.429389208 podStartE2EDuration="2.429389208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:11:57.428054337 +0000 UTC m=+2898.745704337" watchObservedRunningTime="2026-11-01 23:11:57.429389208 +0000 UTC m=+2898.747039221"
Nov 01 23:11:57.551415 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:57.551379    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:58.330972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:58.330933    4531 generic.go:356] "Generic (PLEG): container finished" podID="82193017-a025-4ef2-9f35-460a50a4275c" containerID="3bb4d78a4716957eab983a3d7e73681742c24d46cb1e0494b49e49e6fd375bff" exitCode=0
Nov 01 23:11:58.331455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:58.331022    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5" event={"ID":"82193017-a025-4ef2-9f35-460a50a4275c","Type":"ContainerDied","Data":"3bb4d78a4716957eab983a3d7e73681742c24d46cb1e0494b49e49e6fd375bff"}
Nov 01 23:11:58.750940 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:58.750908    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:11:59.485570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.485540    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:11:59.603236 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.603158    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca\") pod \"82193017-a025-4ef2-9f35-460a50a4275c\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") "
Nov 01 23:11:59.603403 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.603254    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc\" (UniqueName: \"kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc\") pod \"82193017-a025-4ef2-9f35-460a50a4275c\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") "
Nov 01 23:11:59.603403 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.603317    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca\") pod \"82193017-a025-4ef2-9f35-460a50a4275c\" (UID: \"82193017-a025-4ef2-9f35-460a50a4275c\") "
Nov 01 23:11:59.603888 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.603863    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca" pod "82193017-a025-4ef2-9f35-460a50a4275c" (UID: "82193017-a025-4ef2-9f35-460a50a4275c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:11:59.605422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.605389    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca" pod "82193017-a025-4ef2-9f35-460a50a4275c" (UID: "82193017-a025-4ef2-9f35-460a50a4275c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:11:59.605792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.605768    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc" pod "82193017-a025-4ef2-9f35-460a50a4275c" (UID: "82193017-a025-4ef2-9f35-460a50a4275c"). InnerVolumeSpecName "kube-api-access-5hclc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:11:59.704719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.704691    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:59.704719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.704712    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hclc\" (UniqueName: \"kubernetes.io/projected/82193017-a025-4ef2-9f35-460a50a4275c-kube-api-access-5hclc\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:11:59.704901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:11:59.704750    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82193017-a025-4ef2-9f35-460a50a4275c-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:00.347455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:00.347423    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5" event={"ID":"82193017-a025-4ef2-9f35-460a50a4275c","Type":"ContainerDied","Data":"77914ed5981810eddce622177466e6f761e6fdaddad4171b7be85e0956013695"}
Nov 01 23:12:00.347455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:00.347449    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"
Nov 01 23:12:00.347668 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:00.347469    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77914ed5981810eddce622177466e6f761e6fdaddad4171b7be85e0956013695"
Nov 01 23:12:00.642827 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:00.642738    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"]
Nov 01 23:12:00.948794 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:00.948701    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:01.127845 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:01.127810    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:02.134381 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.134346    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"]
Nov 01 23:12:02.134933 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.134919    4531 cpu_manager.go:414] "containerMap: removing container" podUID="82193017-a025-4ef2-9f35-460a50a4275c" containerName="revoke-keys"
Nov 01 23:12:02.135001 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.134935    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82193017-a025-4ef2-9f35-460a50a4275c" containerName="revoke-keys"
Nov 01 23:12:02.135047 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.135033    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="82193017-a025-4ef2-9f35-460a50a4275c" containerName="revoke-keys"
Nov 01 23:12:02.138695 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.138673    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.198937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.198898    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"]
Nov 01 23:12:02.206256 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.205439    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.227380 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.227356    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:12:02.231095 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.231071    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"]
Nov 01 23:12:02.238043 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.238026    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:12:02.244899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.244877    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"]
Nov 01 23:12:02.332443 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.332415    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zcm\" (UniqueName: \"kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.332638 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.332465    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.332638 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.332492    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.332750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.332641    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.332750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.332680    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.332750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.332738    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc4fk\" (UniqueName: \"kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.434064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.433981    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.434064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434035    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.434266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434085    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.434266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434112    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.434266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434165    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc4fk\" (UniqueName: \"kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.434429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434291    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zcm\" (UniqueName: \"kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.434914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434888    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.434914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.434905    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.436615 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.436589    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.436721 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.436621    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.488695 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.488664    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc4fk\" (UniqueName: \"kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk\") pod \"maas-api-revoke-sub-3cb95de2-l2pmj\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.488856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.488670    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zcm\" (UniqueName: \"kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm\") pod \"maas-api-revoke-sub-6b6983c3-lfg9g\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.524204 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.524174    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:02.752331 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.752297    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:02.933595 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:02.933563    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"]
Nov 01 23:12:02.938566 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:12:02.938543    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55ae1d3_2743_4f98_b0d1_5711b4c0950a.slice/crio-0b39a4f57cf1fe38610ebc6632d7103b66e3a5985f0f93b28351a6fe7047322d WatchSource:0}: Error finding container 0b39a4f57cf1fe38610ebc6632d7103b66e3a5985f0f93b28351a6fe7047322d: Status 404 returned error can't find the container with id 0b39a4f57cf1fe38610ebc6632d7103b66e3a5985f0f93b28351a6fe7047322d
Nov 01 23:12:03.129848 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.129770    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"]
Nov 01 23:12:03.133347 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:12:03.133318    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d73b77f_1b2d_41d6_9ddd_73022427d2df.slice/crio-682f4d22750e9ee38556c3e6a172e32d9562c08bc64ae23fe7be8e9907984887 WatchSource:0}: Error finding container 682f4d22750e9ee38556c3e6a172e32d9562c08bc64ae23fe7be8e9907984887: Status 404 returned error can't find the container with id 682f4d22750e9ee38556c3e6a172e32d9562c08bc64ae23fe7be8e9907984887
Nov 01 23:12:03.195942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.195897    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:03.371079 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.371038    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj" event={"ID":"a55ae1d3-2743-4f98-b0d1-5711b4c0950a","Type":"ContainerStarted","Data":"21fce33acb9fda746577af0b79cefad465776c2dc86fabbd1dcebf92c0eaccf1"}
Nov 01 23:12:03.371244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.371096    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj" event={"ID":"a55ae1d3-2743-4f98-b0d1-5711b4c0950a","Type":"ContainerStarted","Data":"0b39a4f57cf1fe38610ebc6632d7103b66e3a5985f0f93b28351a6fe7047322d"}
Nov 01 23:12:03.373134 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.373105    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g" event={"ID":"4d73b77f-1b2d-41d6-9ddd-73022427d2df","Type":"ContainerStarted","Data":"682f4d22750e9ee38556c3e6a172e32d9562c08bc64ae23fe7be8e9907984887"}
Nov 01 23:12:03.424872 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.424798    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg" podUID="49f56cfd-b290-4c3a-9d83-98f3582fb50d" containerName="main" containerID="cri-o://aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66" gracePeriod=45
Nov 01 23:12:03.480085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.480035    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj" podStartSLOduration=1.480020657 podStartE2EDuration="1.480020657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:03.478855004 +0000 UTC m=+2904.796505018" watchObservedRunningTime="2026-11-01 23:12:03.480020657 +0000 UTC m=+2904.797670658"
Nov 01 23:12:03.708891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.708862    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:12:03.748329 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748300    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir\") pod \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") "
Nov 01 23:12:03.748476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748344    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home\") pod \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") "
Nov 01 23:12:03.748476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748377    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache\") pod \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") "
Nov 01 23:12:03.748476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748435    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm\") pod \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") "
Nov 01 23:12:03.748476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748471    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr\" (UniqueName: \"kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr\") pod \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") "
Nov 01 23:12:03.748707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748516    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs\" (UniqueName: \"kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs\") pod \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\" (UID: \"49f56cfd-b290-4c3a-9d83-98f3582fb50d\") "
Nov 01 23:12:03.748707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748529    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir" pod "49f56cfd-b290-4c3a-9d83-98f3582fb50d" (UID: "49f56cfd-b290-4c3a-9d83-98f3582fb50d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:03.748707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748649    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home" pod "49f56cfd-b290-4c3a-9d83-98f3582fb50d" (UID: "49f56cfd-b290-4c3a-9d83-98f3582fb50d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:03.748917 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748785    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache" pod "49f56cfd-b290-4c3a-9d83-98f3582fb50d" (UID: "49f56cfd-b290-4c3a-9d83-98f3582fb50d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:03.748980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748962    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:03.749045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748979    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:03.749045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.748987    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:03.750823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.750793    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr" pod "49f56cfd-b290-4c3a-9d83-98f3582fb50d" (UID: "49f56cfd-b290-4c3a-9d83-98f3582fb50d"). InnerVolumeSpecName "kube-api-access-xvpgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:03.750937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.750867    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs" pod "49f56cfd-b290-4c3a-9d83-98f3582fb50d" (UID: "49f56cfd-b290-4c3a-9d83-98f3582fb50d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:03.750937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.750882    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm" pod "49f56cfd-b290-4c3a-9d83-98f3582fb50d" (UID: "49f56cfd-b290-4c3a-9d83-98f3582fb50d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:12:03.849918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.849883    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/49f56cfd-b290-4c3a-9d83-98f3582fb50d-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:03.849918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.849912    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvpgr\" (UniqueName: \"kubernetes.io/projected/49f56cfd-b290-4c3a-9d83-98f3582fb50d-kube-api-access-xvpgr\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:03.849918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:03.849925    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/49f56cfd-b290-4c3a-9d83-98f3582fb50d-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:04.382762 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.382668    4531 generic.go:356] "Generic (PLEG): container finished" podID="a55ae1d3-2743-4f98-b0d1-5711b4c0950a" containerID="21fce33acb9fda746577af0b79cefad465776c2dc86fabbd1dcebf92c0eaccf1" exitCode=0
Nov 01 23:12:04.383176 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.382755    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj" event={"ID":"a55ae1d3-2743-4f98-b0d1-5711b4c0950a","Type":"ContainerDied","Data":"21fce33acb9fda746577af0b79cefad465776c2dc86fabbd1dcebf92c0eaccf1"}
Nov 01 23:12:04.384423 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.384399    4531 generic.go:356] "Generic (PLEG): container finished" podID="4d73b77f-1b2d-41d6-9ddd-73022427d2df" containerID="7c242fc1e1ca6bf73dc1650122a61942dbf4e843e3e59bc71b895089f9b34125" exitCode=0
Nov 01 23:12:04.384569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.384471    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g" event={"ID":"4d73b77f-1b2d-41d6-9ddd-73022427d2df","Type":"ContainerDied","Data":"7c242fc1e1ca6bf73dc1650122a61942dbf4e843e3e59bc71b895089f9b34125"}
Nov 01 23:12:04.385845 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.385823    4531 generic.go:356] "Generic (PLEG): container finished" podID="49f56cfd-b290-4c3a-9d83-98f3582fb50d" containerID="aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66" exitCode=0
Nov 01 23:12:04.385981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.385861    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"
Nov 01 23:12:04.385981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.385876    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg" event={"ID":"49f56cfd-b290-4c3a-9d83-98f3582fb50d","Type":"ContainerDied","Data":"aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66"}
Nov 01 23:12:04.385981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.385909    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg" event={"ID":"49f56cfd-b290-4c3a-9d83-98f3582fb50d","Type":"ContainerDied","Data":"405477ffe1c6172ce8d6dd59e5a6f13e53d2bd8a2cd04e320e6c3cee139f5065"}
Nov 01 23:12:04.385981 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.385932    4531 scope.go:122] "RemoveContainer" containerID="aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66"
Nov 01 23:12:04.396523 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.396496    4531 scope.go:122] "RemoveContainer" containerID="aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66"
Nov 01 23:12:04.396846 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:12:04.396821    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66\": container with ID starting with aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66 not found: ID does not exist" containerID="aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66"
Nov 01 23:12:04.396936 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.396857    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66"} err="failed to get container status \"aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66\": rpc error: code = NotFound desc = could not find container \"aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66\": container with ID starting with aef8d5696f104c13a8b430fc2dd95c52e1d5e139306ea2b2fba625ff5ddefe66 not found: ID does not exist"
Nov 01 23:12:04.725631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.725599    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"]
Nov 01 23:12:04.728735 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.728706    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-cf46c7/test-model-cf46c7-kserve-6778b5fb5c-sf7tg"]
Nov 01 23:12:04.816892 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:04.816864    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49f56cfd-b290-4c3a-9d83-98f3582fb50d" path="/var/lib/kubelet/pods/49f56cfd-b290-4c3a-9d83-98f3582fb50d/volumes"
Nov 01 23:12:05.566250 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.566222    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:05.568301 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.568284    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:05.643419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.643383    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerName="payload-pre-processing" containerID="cri-o://8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c" gracePeriod=25
Nov 01 23:12:05.667366 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667305    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca\") pod \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") "
Nov 01 23:12:05.667483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667417    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk\" (UniqueName: \"kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk\") pod \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") "
Nov 01 23:12:05.667541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667488    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca\") pod \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") "
Nov 01 23:12:05.667541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667532    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca\") pod \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\" (UID: \"a55ae1d3-2743-4f98-b0d1-5711b4c0950a\") "
Nov 01 23:12:05.667660 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667584    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca\") pod \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") "
Nov 01 23:12:05.667660 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667635    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm\" (UniqueName: \"kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm\") pod \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\" (UID: \"4d73b77f-1b2d-41d6-9ddd-73022427d2df\") "
Nov 01 23:12:05.667810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.667693    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca" pod "a55ae1d3-2743-4f98-b0d1-5711b4c0950a" (UID: "a55ae1d3-2743-4f98-b0d1-5711b4c0950a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:05.668077 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.668042    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca" pod "4d73b77f-1b2d-41d6-9ddd-73022427d2df" (UID: "4d73b77f-1b2d-41d6-9ddd-73022427d2df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:05.668200 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.668184    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:05.668271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.668207    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:05.669604 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.669579    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk" pod "a55ae1d3-2743-4f98-b0d1-5711b4c0950a" (UID: "a55ae1d3-2743-4f98-b0d1-5711b4c0950a"). InnerVolumeSpecName "kube-api-access-vc4fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:05.669698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.669595    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca" pod "4d73b77f-1b2d-41d6-9ddd-73022427d2df" (UID: "4d73b77f-1b2d-41d6-9ddd-73022427d2df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:05.669698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.669643    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca" pod "a55ae1d3-2743-4f98-b0d1-5711b4c0950a" (UID: "a55ae1d3-2743-4f98-b0d1-5711b4c0950a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:05.669698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.669669    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm" pod "4d73b77f-1b2d-41d6-9ddd-73022427d2df" (UID: "4d73b77f-1b2d-41d6-9ddd-73022427d2df"). InnerVolumeSpecName "kube-api-access-r8zcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:05.769368 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.769337    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc4fk\" (UniqueName: \"kubernetes.io/projected/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-kube-api-access-vc4fk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:05.769368 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.769367    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d73b77f-1b2d-41d6-9ddd-73022427d2df-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:05.769570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.769379    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a55ae1d3-2743-4f98-b0d1-5711b4c0950a-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:05.769570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:05.769388    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8zcm\" (UniqueName: \"kubernetes.io/projected/4d73b77f-1b2d-41d6-9ddd-73022427d2df-kube-api-access-r8zcm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:06.401949 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:06.401919    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"
Nov 01 23:12:06.402118 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:06.401952    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj" event={"ID":"a55ae1d3-2743-4f98-b0d1-5711b4c0950a","Type":"ContainerDied","Data":"0b39a4f57cf1fe38610ebc6632d7103b66e3a5985f0f93b28351a6fe7047322d"}
Nov 01 23:12:06.402118 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:06.402001    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b39a4f57cf1fe38610ebc6632d7103b66e3a5985f0f93b28351a6fe7047322d"
Nov 01 23:12:06.403413 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:06.403394    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"
Nov 01 23:12:06.403540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:06.403443    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g" event={"ID":"4d73b77f-1b2d-41d6-9ddd-73022427d2df","Type":"ContainerDied","Data":"682f4d22750e9ee38556c3e6a172e32d9562c08bc64ae23fe7be8e9907984887"}
Nov 01 23:12:06.403540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:06.403485    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682f4d22750e9ee38556c3e6a172e32d9562c08bc64ae23fe7be8e9907984887"
Nov 01 23:12:07.243908 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:07.243875    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"]
Nov 01 23:12:07.355615 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:07.355571    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"]
Nov 01 23:12:07.669916 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:07.669825    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:07.860775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:07.860746    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:08.462165 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:08.462128    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:12.193701 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.193665    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:12:12.244297 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.244244    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf" podUID="29b5e9e9-47f1-42b8-ad1b-a237035b00e0" containerName="payload-processing" containerID="cri-o://9a2ee73f7dbcfc79403c6883c9b2d0e28a337101453f97c603d23dd0ee651be5" gracePeriod=25
Nov 01 23:12:12.356445 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.356413    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf" podUID="2d66bf74-6646-47bc-8291-a39d93c4f047" containerName="payload-pre-processing" containerID="cri-o://a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0" gracePeriod=25
Nov 01 23:12:12.452929 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.452891    4531 generic.go:356] "Generic (PLEG): container finished" podID="29b5e9e9-47f1-42b8-ad1b-a237035b00e0" containerID="9a2ee73f7dbcfc79403c6883c9b2d0e28a337101453f97c603d23dd0ee651be5" exitCode=0
Nov 01 23:12:12.453076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.452964    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf" event={"ID":"29b5e9e9-47f1-42b8-ad1b-a237035b00e0","Type":"ContainerDied","Data":"9a2ee73f7dbcfc79403c6883c9b2d0e28a337101453f97c603d23dd0ee651be5"}
Nov 01 23:12:12.509394 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.509366    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 23:12:12.538739 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:12:12.538687    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:12:12.538940 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.538922    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume\") pod \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") "
Nov 01 23:12:12.539032 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.539019    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df\" (UniqueName: \"kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df\") pod \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\" (UID: \"29b5e9e9-47f1-42b8-ad1b-a237035b00e0\") "
Nov 01 23:12:12.539281 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.539260    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume" pod "29b5e9e9-47f1-42b8-ad1b-a237035b00e0" (UID: "29b5e9e9-47f1-42b8-ad1b-a237035b00e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:12.539464 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.539443    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:12.541388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.541356    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df" pod "29b5e9e9-47f1-42b8-ad1b-a237035b00e0" (UID: "29b5e9e9-47f1-42b8-ad1b-a237035b00e0"). InnerVolumeSpecName "kube-api-access-vr6df". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:12.601826 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.601803    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 23:12:12.640423 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.640397    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4\" (UniqueName: \"kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4\") pod \"2d66bf74-6646-47bc-8291-a39d93c4f047\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") "
Nov 01 23:12:12.640549 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.640527    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume\") pod \"2d66bf74-6646-47bc-8291-a39d93c4f047\" (UID: \"2d66bf74-6646-47bc-8291-a39d93c4f047\") "
Nov 01 23:12:12.640808 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.640792    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr6df\" (UniqueName: \"kubernetes.io/projected/29b5e9e9-47f1-42b8-ad1b-a237035b00e0-kube-api-access-vr6df\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:12.640924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.640896    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume" pod "2d66bf74-6646-47bc-8291-a39d93c4f047" (UID: "2d66bf74-6646-47bc-8291-a39d93c4f047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:12.642540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.642513    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4" pod "2d66bf74-6646-47bc-8291-a39d93c4f047" (UID: "2d66bf74-6646-47bc-8291-a39d93c4f047"). InnerVolumeSpecName "kube-api-access-dc5x4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:12.741543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.741475    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dc5x4\" (UniqueName: \"kubernetes.io/projected/2d66bf74-6646-47bc-8291-a39d93c4f047-kube-api-access-dc5x4\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:12.741543 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:12.741505    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d66bf74-6646-47bc-8291-a39d93c4f047-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:13.465075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.465041    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"
Nov 01 23:12:13.465567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.465042    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf" event={"ID":"29b5e9e9-47f1-42b8-ad1b-a237035b00e0","Type":"ContainerDied","Data":"c598bbc0b872ea83f1ace179ba80fcbf4b4cb855e7f06d58be6adc360ef25eda"}
Nov 01 23:12:13.465567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.465186    4531 scope.go:122] "RemoveContainer" containerID="9a2ee73f7dbcfc79403c6883c9b2d0e28a337101453f97c603d23dd0ee651be5"
Nov 01 23:12:13.466864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.466809    4531 generic.go:356] "Generic (PLEG): container finished" podID="2d66bf74-6646-47bc-8291-a39d93c4f047" containerID="a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0" exitCode=0
Nov 01 23:12:13.466864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.466859    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"
Nov 01 23:12:13.467044 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.466979    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf" event={"ID":"2d66bf74-6646-47bc-8291-a39d93c4f047","Type":"ContainerDied","Data":"a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0"}
Nov 01 23:12:13.467044 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.467033    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf" event={"ID":"2d66bf74-6646-47bc-8291-a39d93c4f047","Type":"ContainerDied","Data":"1b15b25e62e4a3b77ff61b36d8ae270c4f8830a87e6621f1f7c69071db74d6dd"}
Nov 01 23:12:13.476016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.475995    4531 scope.go:122] "RemoveContainer" containerID="a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0"
Nov 01 23:12:13.485094 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.485076    4531 scope.go:122] "RemoveContainer" containerID="a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0"
Nov 01 23:12:13.485350 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:12:13.485331    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0\": container with ID starting with a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0 not found: ID does not exist" containerID="a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0"
Nov 01 23:12:13.485424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.485360    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0"} err="failed to get container status \"a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0\": rpc error: code = NotFound desc = could not find container \"a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0\": container with ID starting with a57a4020f7b7715fee8b9df5f307636943c41e692f09b68dabba567bafa297c0 not found: ID does not exist"
Nov 01 23:12:13.521324 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.521272    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:13.626325 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.626295    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"]
Nov 01 23:12:13.670525 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.670500    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-646881-5496cbb95f-xhcmf"]
Nov 01 23:12:13.772779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.772712    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"]
Nov 01 23:12:13.778420 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:13.778398    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-646881-7d944bfbb7-fpbmf"]
Nov 01 23:12:14.816088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:14.816052    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b5e9e9-47f1-42b8-ad1b-a237035b00e0" path="/var/lib/kubelet/pods/29b5e9e9-47f1-42b8-ad1b-a237035b00e0/volumes"
Nov 01 23:12:14.816590 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:14.816574    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d66bf74-6646-47bc-8291-a39d93c4f047" path="/var/lib/kubelet/pods/2d66bf74-6646-47bc-8291-a39d93c4f047/volumes"
Nov 01 23:12:18.241242 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:18.241187    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:22.193404 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:22.193310    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:12:23.148483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:23.148448    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:23.480020 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:23.479985    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:23.675169 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:23.675132    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:25.920293 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:25.920264    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:12:26.075789 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.075764    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf\" (UniqueName: \"kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf\") pod \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") "
Nov 01 23:12:26.075963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.075919    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume\") pod \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\" (UID: \"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f\") "
Nov 01 23:12:26.076319 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.076294    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume" pod "b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" (UID: "b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:26.077835 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.077807    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf" pod "b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" (UID: "b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f"). InnerVolumeSpecName "kube-api-access-fhdvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:26.176983 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.176927    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:26.176983 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.176950    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdvf\" (UniqueName: \"kubernetes.io/projected/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f-kube-api-access-fhdvf\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:26.574704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.574666    4531 generic.go:356] "Generic (PLEG): container finished" podID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerID="8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c" exitCode=0
Nov 01 23:12:26.574884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.574714    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"
Nov 01 23:12:26.574884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.574749    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" event={"ID":"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f","Type":"ContainerDied","Data":"8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c"}
Nov 01 23:12:26.574884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.574805    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc" event={"ID":"b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f","Type":"ContainerDied","Data":"de1745ddee242853c73ea994c654942d3cbd541620c7d19090756850c8196ff3"}
Nov 01 23:12:26.574884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.574828    4531 scope.go:122] "RemoveContainer" containerID="8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c"
Nov 01 23:12:26.585604 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.585549    4531 scope.go:122] "RemoveContainer" containerID="8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c"
Nov 01 23:12:26.585966 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:12:26.585904    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c\": container with ID starting with 8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c not found: ID does not exist" containerID="8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c"
Nov 01 23:12:26.585966 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.585938    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c"} err="failed to get container status \"8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c\": rpc error: code = NotFound desc = could not find container \"8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c\": container with ID starting with 8680eb266d8c01778608e49e7d30728f9ae59088bd6ac8c0293efc04c9054f2c not found: ID does not exist"
Nov 01 23:12:26.734528 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.734495    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"]
Nov 01 23:12:26.777934 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.777902    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aa47d8-8478cbdc59-wm2rc"]
Nov 01 23:12:26.816033 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:26.816003    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" path="/var/lib/kubelet/pods/b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f/volumes"
Nov 01 23:12:27.320719 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:27.320687    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:28.830584 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:28.830550    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:29.065451 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:29.065419    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:29.264012 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:29.263966    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:29.325319 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:29.325292    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"]
Nov 01 23:12:29.330455 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:29.330431    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aa47d8-l22ks"]
Nov 01 23:12:29.725484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:29.725454    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:30.817067 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:30.817035    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f87df9-3e0b-4ceb-8de4-82ab72289f19" path="/var/lib/kubelet/pods/b1f87df9-3e0b-4ceb-8de4-82ab72289f19/volumes"
Nov 01 23:12:34.169552 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:34.169512    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:36.214476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:36.214442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:36.428298 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:36.428261    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:36.630760 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:36.630637    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:37.341358 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341324    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"]
Nov 01 23:12:37.341862 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341847    4531 cpu_manager.go:414] "containerMap: removing container" podUID="29b5e9e9-47f1-42b8-ad1b-a237035b00e0" containerName="payload-processing"
Nov 01 23:12:37.341914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341863    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b5e9e9-47f1-42b8-ad1b-a237035b00e0" containerName="payload-processing"
Nov 01 23:12:37.341914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341874    4531 cpu_manager.go:414] "containerMap: removing container" podUID="4d73b77f-1b2d-41d6-9ddd-73022427d2df" containerName="revoke-keys"
Nov 01 23:12:37.341914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341879    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d73b77f-1b2d-41d6-9ddd-73022427d2df" containerName="revoke-keys"
Nov 01 23:12:37.341914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341902    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a55ae1d3-2743-4f98-b0d1-5711b4c0950a" containerName="revoke-keys"
Nov 01 23:12:37.341914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341907    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55ae1d3-2743-4f98-b0d1-5711b4c0950a" containerName="revoke-keys"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341917    4531 cpu_manager.go:414] "containerMap: removing container" podUID="49f56cfd-b290-4c3a-9d83-98f3582fb50d" containerName="main"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341922    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49f56cfd-b290-4c3a-9d83-98f3582fb50d" containerName="main"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341934    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2d66bf74-6646-47bc-8291-a39d93c4f047" containerName="payload-pre-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341938    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d66bf74-6646-47bc-8291-a39d93c4f047" containerName="payload-pre-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341953    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerName="payload-pre-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.341957    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerName="payload-pre-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.342031    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55ae1d3-2743-4f98-b0d1-5711b4c0950a" containerName="revoke-keys"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.342041    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="49f56cfd-b290-4c3a-9d83-98f3582fb50d" containerName="main"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.342050    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d66bf74-6646-47bc-8291-a39d93c4f047" containerName="payload-pre-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.342058    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2af3b9b-1f8a-4ecd-b3bf-16f5ee1f8d6f" containerName="payload-pre-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.342066    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b5e9e9-47f1-42b8-ad1b-a237035b00e0" containerName="payload-processing"
Nov 01 23:12:37.342080 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.342072    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d73b77f-1b2d-41d6-9ddd-73022427d2df" containerName="revoke-keys"
Nov 01 23:12:37.346513 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.346495    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.441879 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.441846    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:12:37.442054 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.441946    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:12:37.445327 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.445306    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"]
Nov 01 23:12:37.493816 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.493783    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.493989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.493937    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.494035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.494007    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtd56\" (UniqueName: \"kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.594658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.594582    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.594658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.594643    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtd56\" (UniqueName: \"kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.594915 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.594745    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.595537 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.595514    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.597204 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.597179    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.649024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.648998    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtd56\" (UniqueName: \"kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56\") pod \"maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:37.660104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:37.660082    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:38.038551 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:38.038513    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"]
Nov 01 23:12:38.043034 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:12:38.043007    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1596de7b_1bce_4ebd_8ec0_d8263ccdaa66.slice/crio-68291c0150d738d3c39301bd57c7cc96d9a12cd2aabc5b42c0efab3c3d8c7a30 WatchSource:0}: Error finding container 68291c0150d738d3c39301bd57c7cc96d9a12cd2aabc5b42c0efab3c3d8c7a30: Status 404 returned error can't find the container with id 68291c0150d738d3c39301bd57c7cc96d9a12cd2aabc5b42c0efab3c3d8c7a30
Nov 01 23:12:38.668475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:38.668394    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7" event={"ID":"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66","Type":"ContainerStarted","Data":"10e48b6dc278e8cbfe49e78ace030e22495dd8f632afa1f97d14209ed90323a4"}
Nov 01 23:12:38.668475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:38.668443    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7" event={"ID":"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66","Type":"ContainerStarted","Data":"68291c0150d738d3c39301bd57c7cc96d9a12cd2aabc5b42c0efab3c3d8c7a30"}
Nov 01 23:12:38.777958 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:38.777913    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7" podStartSLOduration=1.777898361 podStartE2EDuration="1.777898361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:38.775828459 +0000 UTC m=+2940.093478462" watchObservedRunningTime="2026-11-01 23:12:38.777898361 +0000 UTC m=+2940.095548353"
Nov 01 23:12:39.376973 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:39.376890    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:39.677483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:39.677410    4531 generic.go:356] "Generic (PLEG): container finished" podID="1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" containerID="10e48b6dc278e8cbfe49e78ace030e22495dd8f632afa1f97d14209ed90323a4" exitCode=0
Nov 01 23:12:39.677483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:39.677450    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7" event={"ID":"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66","Type":"ContainerDied","Data":"10e48b6dc278e8cbfe49e78ace030e22495dd8f632afa1f97d14209ed90323a4"}
Nov 01 23:12:40.835197 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.835175    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:40.929024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.928996    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca\") pod \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") "
Nov 01 23:12:40.929161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.929056    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca\") pod \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") "
Nov 01 23:12:40.929161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.929126    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56\" (UniqueName: \"kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56\") pod \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\" (UID: \"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66\") "
Nov 01 23:12:40.929375 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.929351    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca" pod "1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" (UID: "1596de7b-1bce-4ebd-8ec0-d8263ccdaa66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:40.929473 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.929457    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:40.931200 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.931178    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca" pod "1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" (UID: "1596de7b-1bce-4ebd-8ec0-d8263ccdaa66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:40.931281 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:40.931196    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56" pod "1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" (UID: "1596de7b-1bce-4ebd-8ec0-d8263ccdaa66"). InnerVolumeSpecName "kube-api-access-jtd56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:41.030192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:41.030167    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.030192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:41.030191    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtd56\" (UniqueName: \"kubernetes.io/projected/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66-kube-api-access-jtd56\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:41.693887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:41.693848    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"
Nov 01 23:12:41.693887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:41.693871    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7" event={"ID":"1596de7b-1bce-4ebd-8ec0-d8263ccdaa66","Type":"ContainerDied","Data":"68291c0150d738d3c39301bd57c7cc96d9a12cd2aabc5b42c0efab3c3d8c7a30"}
Nov 01 23:12:41.694120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:41.693916    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68291c0150d738d3c39301bd57c7cc96d9a12cd2aabc5b42c0efab3c3d8c7a30"
Nov 01 23:12:42.525873 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:12:42.525795    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:12:43.655221 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:43.655191    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:44.405447 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:44.405413    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:44.792260 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:44.792228    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:45.170384 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:45.170308    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:45.437321 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:45.437240    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:45.604516 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:45.604484    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"]
Nov 01 23:12:45.650567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:45.650535    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-a6dd2f-ltrjj"]
Nov 01 23:12:46.097370 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.097340    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:46.354132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.354045    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:46.623096 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.623020    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:46.808078 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.808048    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:46.816383 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.816357    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87" path="/var/lib/kubelet/pods/7b7a2f0c-658f-4f3f-9eb3-37c7d96d9b87/volumes"
Nov 01 23:12:46.841142 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.841118    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"]
Nov 01 23:12:46.889076 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:46.889008    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-646881-mbsv5"]
Nov 01 23:12:47.332334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:47.332301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:48.818052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:48.818024    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82193017-a025-4ef2-9f35-460a50a4275c" path="/var/lib/kubelet/pods/82193017-a025-4ef2-9f35-460a50a4275c/volumes"
Nov 01 23:12:48.916337 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:48.916310    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"]
Nov 01 23:12:49.020519 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:49.020491    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"]
Nov 01 23:12:49.330199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:49.330151    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:49.510717 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:49.510685    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:51.919942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:51.919906    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:52.122648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:52.122616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:52.330337 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:52.330301    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:52.504533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:52.504496    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:53.610878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:53.610836    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:53.807084 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:53.807047    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:53.917551 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:53.917447    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerName="payload-processing" containerID="cri-o://c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea" gracePeriod=25
Nov 01 23:12:53.989575 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:53.989536    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:54.020906 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.020876    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerName="payload-pre-processing" containerID="cri-o://4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2" gracePeriod=25
Nov 01 23:12:54.737209 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.737175    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"]
Nov 01 23:12:54.737700 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.737686    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" containerName="revoke-keys"
Nov 01 23:12:54.737770 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.737700    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" containerName="revoke-keys"
Nov 01 23:12:54.737828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.737817    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" containerName="revoke-keys"
Nov 01 23:12:54.742909 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.742892    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.796856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.796827    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"]
Nov 01 23:12:54.837489 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.837458    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:12:54.837489 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.837476    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:12:54.883342 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.883307    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.883539 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.883409    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.883605 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.883579    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbj9s\" (UniqueName: \"kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.984698 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.984669    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.984891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.984768    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbj9s\" (UniqueName: \"kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.984891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.984822    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.985407 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.985380    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:54.987164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:54.987147    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:55.038567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:55.038517    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbj9s\" (UniqueName: \"kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:55.056978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:55.056957    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:55.433164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:55.433079    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"]
Nov 01 23:12:55.438071 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:12:55.438042    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4921d3a_56d8_40db_a7ee_5039b886817b.slice/crio-87ba626720df76a5a8c43d4f4ce1c8edc10b88ed05e131a3027708b3c11e18ef WatchSource:0}: Error finding container 87ba626720df76a5a8c43d4f4ce1c8edc10b88ed05e131a3027708b3c11e18ef: Status 404 returned error can't find the container with id 87ba626720df76a5a8c43d4f4ce1c8edc10b88ed05e131a3027708b3c11e18ef
Nov 01 23:12:55.814048 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:55.814009    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj" event={"ID":"e4921d3a-56d8-40db-a7ee-5039b886817b","Type":"ContainerStarted","Data":"c260e74c51240413922f0c4c87ce19acdfc673100907b26856f2f5e449a28e8d"}
Nov 01 23:12:55.814385 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:55.814072    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj" event={"ID":"e4921d3a-56d8-40db-a7ee-5039b886817b","Type":"ContainerStarted","Data":"87ba626720df76a5a8c43d4f4ce1c8edc10b88ed05e131a3027708b3c11e18ef"}
Nov 01 23:12:55.922089 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:55.922039    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj" podStartSLOduration=1.9220251940000002 podStartE2EDuration="1.922025194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:12:55.919780634 +0000 UTC m=+2957.237430630" watchObservedRunningTime="2026-11-01 23:12:55.922025194 +0000 UTC m=+2957.239675195"
Nov 01 23:12:56.822003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:56.821971    4531 generic.go:356] "Generic (PLEG): container finished" podID="e4921d3a-56d8-40db-a7ee-5039b886817b" containerID="c260e74c51240413922f0c4c87ce19acdfc673100907b26856f2f5e449a28e8d" exitCode=0
Nov 01 23:12:56.822376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:56.822035    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj" event={"ID":"e4921d3a-56d8-40db-a7ee-5039b886817b","Type":"ContainerDied","Data":"c260e74c51240413922f0c4c87ce19acdfc673100907b26856f2f5e449a28e8d"}
Nov 01 23:12:56.966863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:56.966828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:12:57.979191 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:57.979157    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:12:58.115038 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.114940    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca\") pod \"e4921d3a-56d8-40db-a7ee-5039b886817b\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") "
Nov 01 23:12:58.115216 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.115119    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s\" (UniqueName: \"kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s\") pod \"e4921d3a-56d8-40db-a7ee-5039b886817b\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") "
Nov 01 23:12:58.115216 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.115158    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca\") pod \"e4921d3a-56d8-40db-a7ee-5039b886817b\" (UID: \"e4921d3a-56d8-40db-a7ee-5039b886817b\") "
Nov 01 23:12:58.115568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.115537    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca" pod "e4921d3a-56d8-40db-a7ee-5039b886817b" (UID: "e4921d3a-56d8-40db-a7ee-5039b886817b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:12:58.115718 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.115693    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:58.117127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.117100    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca" pod "e4921d3a-56d8-40db-a7ee-5039b886817b" (UID: "e4921d3a-56d8-40db-a7ee-5039b886817b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:12:58.117518 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.117500    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s" pod "e4921d3a-56d8-40db-a7ee-5039b886817b" (UID: "e4921d3a-56d8-40db-a7ee-5039b886817b"). InnerVolumeSpecName "kube-api-access-fbj9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:12:58.217151 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.217119    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4921d3a-56d8-40db-a7ee-5039b886817b-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:58.217151 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.217149    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbj9s\" (UniqueName: \"kubernetes.io/projected/e4921d3a-56d8-40db-a7ee-5039b886817b-kube-api-access-fbj9s\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:12:58.838305 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.838271    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj" event={"ID":"e4921d3a-56d8-40db-a7ee-5039b886817b","Type":"ContainerDied","Data":"87ba626720df76a5a8c43d4f4ce1c8edc10b88ed05e131a3027708b3c11e18ef"}
Nov 01 23:12:58.838444 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.838319    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87ba626720df76a5a8c43d4f4ce1c8edc10b88ed05e131a3027708b3c11e18ef"
Nov 01 23:12:58.838444 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:12:58.838285    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"
Nov 01 23:13:00.004459 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:00.004422    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:00.004935 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:00.004428    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:02.124860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:02.124816    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:04.728864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:04.728827    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:06.641713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:06.641678    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:06.823972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:06.823939    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:10.004642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:10.004609    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:10.004642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:10.004634    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:10.971714 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:10.971679    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:12.486524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:12.486486    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:12.526317 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:13:12.526283    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:13:12.724624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:12.724582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:12.930153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:12.930074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:12.938717 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:12.938686    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"]
Nov 01 23:13:12.984924 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:12.984898    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cf46c7-464l7"]
Nov 01 23:13:13.420863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:13.420818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:14.191990 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.191931    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:13:14.276930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.276906    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:13:14.282127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.282108    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv\" (UniqueName: \"kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv\") pod \"52362a8f-e57d-41dc-805c-bcfd024f143d\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") "
Nov 01 23:13:14.282214 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.282201    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume\") pod \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") "
Nov 01 23:13:14.282266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.282255    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume\") pod \"52362a8f-e57d-41dc-805c-bcfd024f143d\" (UID: \"52362a8f-e57d-41dc-805c-bcfd024f143d\") "
Nov 01 23:13:14.282306 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.282283    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z\" (UniqueName: \"kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z\") pod \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\" (UID: \"0b7bb7df-9c18-4e39-aa88-606a4da1a139\") "
Nov 01 23:13:14.282556 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.282533    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume" pod "0b7bb7df-9c18-4e39-aa88-606a4da1a139" (UID: "0b7bb7df-9c18-4e39-aa88-606a4da1a139"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:14.282630 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.282607    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume" pod "52362a8f-e57d-41dc-805c-bcfd024f143d" (UID: "52362a8f-e57d-41dc-805c-bcfd024f143d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:14.284084 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.284066    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv" pod "52362a8f-e57d-41dc-805c-bcfd024f143d" (UID: "52362a8f-e57d-41dc-805c-bcfd024f143d"). InnerVolumeSpecName "kube-api-access-pjdfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:14.284241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.284219    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z" pod "0b7bb7df-9c18-4e39-aa88-606a4da1a139" (UID: "0b7bb7df-9c18-4e39-aa88-606a4da1a139"). InnerVolumeSpecName "kube-api-access-mbf4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:14.382939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.382912    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52362a8f-e57d-41dc-805c-bcfd024f143d-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:14.382939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.382941    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbf4z\" (UniqueName: \"kubernetes.io/projected/0b7bb7df-9c18-4e39-aa88-606a4da1a139-kube-api-access-mbf4z\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:14.383130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.382955    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdfv\" (UniqueName: \"kubernetes.io/projected/52362a8f-e57d-41dc-805c-bcfd024f143d-kube-api-access-pjdfv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:14.383130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.382974    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b7bb7df-9c18-4e39-aa88-606a4da1a139-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:14.816706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.816678    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1596de7b-1bce-4ebd-8ec0-d8263ccdaa66" path="/var/lib/kubelet/pods/1596de7b-1bce-4ebd-8ec0-d8263ccdaa66/volumes"
Nov 01 23:13:14.950623 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.950595    4531 generic.go:356] "Generic (PLEG): container finished" podID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerID="c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea" exitCode=0
Nov 01 23:13:14.950771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.950653    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"
Nov 01 23:13:14.950771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.950683    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" event={"ID":"52362a8f-e57d-41dc-805c-bcfd024f143d","Type":"ContainerDied","Data":"c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea"}
Nov 01 23:13:14.950771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.950766    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq" event={"ID":"52362a8f-e57d-41dc-805c-bcfd024f143d","Type":"ContainerDied","Data":"e6ce563f05641047335f352608a23a220f775a02bcda3864426f6c41d976e167"}
Nov 01 23:13:14.950952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.950798    4531 scope.go:122] "RemoveContainer" containerID="c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea"
Nov 01 23:13:14.953127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.953105    4531 generic.go:356] "Generic (PLEG): container finished" podID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerID="4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2" exitCode=0
Nov 01 23:13:14.953236 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.953221    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" event={"ID":"0b7bb7df-9c18-4e39-aa88-606a4da1a139","Type":"ContainerDied","Data":"4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2"}
Nov 01 23:13:14.953305 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.953261    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf" event={"ID":"0b7bb7df-9c18-4e39-aa88-606a4da1a139","Type":"ContainerDied","Data":"e2782ad048d835833bdc5446be6fb0b8922010a4e2d43360a457e0c668bf7cc4"}
Nov 01 23:13:14.953305 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.953283    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"
Nov 01 23:13:14.961722 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.961705    4531 scope.go:122] "RemoveContainer" containerID="c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea"
Nov 01 23:13:14.962041 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:13:14.962019    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea\": container with ID starting with c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea not found: ID does not exist" containerID="c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea"
Nov 01 23:13:14.962101 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.962052    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea"} err="failed to get container status \"c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea\": rpc error: code = NotFound desc = could not find container \"c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea\": container with ID starting with c91315a2b8069e8c6e53ba9a16b6539ac458db33451593227a21bd49df6f5fea not found: ID does not exist"
Nov 01 23:13:14.962101 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.962075    4531 scope.go:122] "RemoveContainer" containerID="4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2"
Nov 01 23:13:14.972583 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.972569    4531 scope.go:122] "RemoveContainer" containerID="4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2"
Nov 01 23:13:14.972892 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:13:14.972873    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2\": container with ID starting with 4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2 not found: ID does not exist" containerID="4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2"
Nov 01 23:13:14.972978 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:14.972899    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2"} err="failed to get container status \"4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2\": rpc error: code = NotFound desc = could not find container \"4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2\": container with ID starting with 4e82e9988de874e93a43110f5a7700c359589d4d05b844522dbd7d115ad195c2 not found: ID does not exist"
Nov 01 23:13:15.106557 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:15.106488    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"]
Nov 01 23:13:15.111531 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:15.111510    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cf46c7-55c9cdc79-s5gzf"]
Nov 01 23:13:15.213424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:15.213397    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"]
Nov 01 23:13:15.216462 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:15.216435    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cf46c7-7784fc944f-vprsq"]
Nov 01 23:13:16.816268 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:16.816236    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" path="/var/lib/kubelet/pods/0b7bb7df-9c18-4e39-aa88-606a4da1a139/volumes"
Nov 01 23:13:16.816675 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:16.816620    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" path="/var/lib/kubelet/pods/52362a8f-e57d-41dc-805c-bcfd024f143d/volumes"
Nov 01 23:13:17.249374 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:17.249338    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:18.767529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:18.767495    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:19.038104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:19.038029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:19.234300 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:19.234245    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"]
Nov 01 23:13:19.271253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:19.271222    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:19.280826 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:19.280761    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-8cc47a-s58nj"]
Nov 01 23:13:19.684431 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:19.684391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:19.888915 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:19.888869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:20.148971 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:20.148940    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:20.815922 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:20.815890    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4921d3a-56d8-40db-a7ee-5039b886817b" path="/var/lib/kubelet/pods/e4921d3a-56d8-40db-a7ee-5039b886817b/volumes"
Nov 01 23:13:24.171429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:24.171385    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:25.575330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:25.575259    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:25.766124 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:25.766090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:25.965178 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:25.965126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:26.369452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:26.369335    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:29.459492 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:29.459377    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:31.286739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:31.286696    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"]
Nov 01 23:13:31.332141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:31.332109    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"]
Nov 01 23:13:31.440919 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:31.440884    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:32.849189 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:32.849124    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:33.034095 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:33.034058    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:34.660027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:34.659982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:38.333689 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:38.333651    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"]
Nov 01 23:13:38.441646 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:38.441610    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"]
Nov 01 23:13:38.756203 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:38.756169    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:38.929604 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:38.929572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:39.756897 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:39.756856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:42.241047 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:42.241018    4531 scope.go:122] "RemoveContainer" containerID="54a506848dfb3b6b76fbbbaa6d3096f0fdd5329104afda4e18627268d50d7340"
Nov 01 23:13:42.525970 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:13:42.525889    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:13:43.334674 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:43.334639    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerName="payload-processing" containerID="cri-o://2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547" gracePeriod=25
Nov 01 23:13:43.442610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:43.442579    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerName="payload-pre-processing" containerID="cri-o://c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103" gracePeriod=25
Nov 01 23:13:44.142400 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:44.142351    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:44.750567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:44.750531    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:45.660046 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:45.660007    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:45.884567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:45.884528    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:46.115149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.115112    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:46.307634 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.307594    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main" containerID="cri-o://a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b" gracePeriod=45
Nov 01 23:13:46.351283 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.351248    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main" containerID="cri-o://93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d" gracePeriod=45
Nov 01 23:13:46.573514 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.573473    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:46.611669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.611641    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 23:13:46.637868 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.637848    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 23:13:46.701995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.701957    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache\") pod \"47f40cac-b1dc-407e-baed-3256eb2f0699\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") "
Nov 01 23:13:46.701995 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.701995    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv\" (UniqueName: \"kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv\") pod \"47f40cac-b1dc-407e-baed-3256eb2f0699\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") "
Nov 01 23:13:46.702236 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.702048    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm\") pod \"47f40cac-b1dc-407e-baed-3256eb2f0699\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") "
Nov 01 23:13:46.702299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.702243    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache" pod "47f40cac-b1dc-407e-baed-3256eb2f0699" (UID: "47f40cac-b1dc-407e-baed-3256eb2f0699"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.702469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.702445    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home\") pod \"47f40cac-b1dc-407e-baed-3256eb2f0699\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") "
Nov 01 23:13:46.702613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.702532    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs\" (UniqueName: \"kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs\") pod \"47f40cac-b1dc-407e-baed-3256eb2f0699\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") "
Nov 01 23:13:46.702613 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.702590    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir\") pod \"47f40cac-b1dc-407e-baed-3256eb2f0699\" (UID: \"47f40cac-b1dc-407e-baed-3256eb2f0699\") "
Nov 01 23:13:46.703262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.703037    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home" pod "47f40cac-b1dc-407e-baed-3256eb2f0699" (UID: "47f40cac-b1dc-407e-baed-3256eb2f0699"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.703262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.703171    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.703262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.703239    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.703262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.703245    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir" pod "47f40cac-b1dc-407e-baed-3256eb2f0699" (UID: "47f40cac-b1dc-407e-baed-3256eb2f0699"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.704823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.704781    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm" pod "47f40cac-b1dc-407e-baed-3256eb2f0699" (UID: "47f40cac-b1dc-407e-baed-3256eb2f0699"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.705537 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.705511    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs" pod "47f40cac-b1dc-407e-baed-3256eb2f0699" (UID: "47f40cac-b1dc-407e-baed-3256eb2f0699"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:46.705713 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.705688    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv" pod "47f40cac-b1dc-407e-baed-3256eb2f0699" (UID: "47f40cac-b1dc-407e-baed-3256eb2f0699"). InnerVolumeSpecName "kube-api-access-czcgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:46.804315 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804282    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache\") pod \"39ebfba5-06df-4469-8a21-8fd8f562dcea\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") "
Nov 01 23:13:46.804506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804354    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65\" (UniqueName: \"kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65\") pod \"39ebfba5-06df-4469-8a21-8fd8f562dcea\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") "
Nov 01 23:13:46.804506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804463    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir\") pod \"39ebfba5-06df-4469-8a21-8fd8f562dcea\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") "
Nov 01 23:13:46.804641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804563    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs\" (UniqueName: \"kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs\") pod \"39ebfba5-06df-4469-8a21-8fd8f562dcea\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") "
Nov 01 23:13:46.804641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804583    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache" pod "39ebfba5-06df-4469-8a21-8fd8f562dcea" (UID: "39ebfba5-06df-4469-8a21-8fd8f562dcea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.804775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804641    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm\") pod \"39ebfba5-06df-4469-8a21-8fd8f562dcea\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") "
Nov 01 23:13:46.804775 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804677    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home\") pod \"39ebfba5-06df-4469-8a21-8fd8f562dcea\" (UID: \"39ebfba5-06df-4469-8a21-8fd8f562dcea\") "
Nov 01 23:13:46.804911 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804800    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir" pod "39ebfba5-06df-4469-8a21-8fd8f562dcea" (UID: "39ebfba5-06df-4469-8a21-8fd8f562dcea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.805015 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.804994    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home" pod "39ebfba5-06df-4469-8a21-8fd8f562dcea" (UID: "39ebfba5-06df-4469-8a21-8fd8f562dcea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805098    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/47f40cac-b1dc-407e-baed-3256eb2f0699-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805119    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805133    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805149    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czcgv\" (UniqueName: \"kubernetes.io/projected/47f40cac-b1dc-407e-baed-3256eb2f0699-kube-api-access-czcgv\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805162    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/47f40cac-b1dc-407e-baed-3256eb2f0699-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805174    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.805208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.805187    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.807071 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.807044    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65" pod "39ebfba5-06df-4469-8a21-8fd8f562dcea" (UID: "39ebfba5-06df-4469-8a21-8fd8f562dcea"). InnerVolumeSpecName "kube-api-access-k6d65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:46.807314 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.807288    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm" pod "39ebfba5-06df-4469-8a21-8fd8f562dcea" (UID: "39ebfba5-06df-4469-8a21-8fd8f562dcea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:13:46.807420 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.807338    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs" pod "39ebfba5-06df-4469-8a21-8fd8f562dcea" (UID: "39ebfba5-06df-4469-8a21-8fd8f562dcea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:46.906631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.906547    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/39ebfba5-06df-4469-8a21-8fd8f562dcea-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.906631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.906575    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6d65\" (UniqueName: \"kubernetes.io/projected/39ebfba5-06df-4469-8a21-8fd8f562dcea-kube-api-access-k6d65\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:46.906631 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:46.906587    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39ebfba5-06df-4469-8a21-8fd8f562dcea-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:47.193965 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.193872    4531 generic.go:356] "Generic (PLEG): container finished" podID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerID="a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b" exitCode=0
Nov 01 23:13:47.193965 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.193943    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"
Nov 01 23:13:47.194208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.193956    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" event={"ID":"47f40cac-b1dc-407e-baed-3256eb2f0699","Type":"ContainerDied","Data":"a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b"}
Nov 01 23:13:47.194208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.194003    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd" event={"ID":"47f40cac-b1dc-407e-baed-3256eb2f0699","Type":"ContainerDied","Data":"61f9dbd9c7616ee688cc7c01a682da14a151da3e09013a54e7b58203513c7742"}
Nov 01 23:13:47.194208 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.194024    4531 scope.go:122] "RemoveContainer" containerID="a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b"
Nov 01 23:13:47.195966 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.195940    4531 generic.go:356] "Generic (PLEG): container finished" podID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerID="93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d" exitCode=0
Nov 01 23:13:47.196114 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.195985    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"
Nov 01 23:13:47.196114 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.196041    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" event={"ID":"39ebfba5-06df-4469-8a21-8fd8f562dcea","Type":"ContainerDied","Data":"93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d"}
Nov 01 23:13:47.196114 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.196071    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5" event={"ID":"39ebfba5-06df-4469-8a21-8fd8f562dcea","Type":"ContainerDied","Data":"1b3436e54e30de42b65b968ccd180920340db58ace6170553b20f57e96fbf677"}
Nov 01 23:13:47.205434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.205403    4531 scope.go:122] "RemoveContainer" containerID="a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b"
Nov 01 23:13:47.205788 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:13:47.205763    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b\": container with ID starting with a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b not found: ID does not exist" containerID="a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b"
Nov 01 23:13:47.205864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.205798    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b"} err="failed to get container status \"a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b\": rpc error: code = NotFound desc = could not find container \"a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b\": container with ID starting with a7df3ff345d929a988a7ec21f2576e208d7379eabcee00d82cce2eea40da349b not found: ID does not exist"
Nov 01 23:13:47.205864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.205816    4531 scope.go:122] "RemoveContainer" containerID="93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d"
Nov 01 23:13:47.218301 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.215483    4531 scope.go:122] "RemoveContainer" containerID="93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d"
Nov 01 23:13:47.218606 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:13:47.218583    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d\": container with ID starting with 93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d not found: ID does not exist" containerID="93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d"
Nov 01 23:13:47.218654 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.218611    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d"} err="failed to get container status \"93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d\": rpc error: code = NotFound desc = could not find container \"93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d\": container with ID starting with 93b2f207d3f6545b03f4b24431c10addb05fe50d228e0ef69597bdcca5b6724d not found: ID does not exist"
Nov 01 23:13:47.348892 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.348858    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"]
Nov 01 23:13:47.354448 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.354421    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/distinct-w2-c768f3-kserve-978cd7c68-85qqd"]
Nov 01 23:13:47.499217 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.499182    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"]
Nov 01 23:13:47.502911 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:47.502887    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c768f3/simulator-w2-c768f3-kserve-6966f965f6-bcpf5"]
Nov 01 23:13:48.816364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:48.816328    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" path="/var/lib/kubelet/pods/39ebfba5-06df-4469-8a21-8fd8f562dcea/volumes"
Nov 01 23:13:48.816783 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:48.816747    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" path="/var/lib/kubelet/pods/47f40cac-b1dc-407e-baed-3256eb2f0699/volumes"
Nov 01 23:13:49.409932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:49.409844    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:49.409932 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:49.409879    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:51.109539 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:51.109500    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:52.325413 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:52.325275    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:52.515709 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:52.515656    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:52.702050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:52.702013    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:55.473258 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473224    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"]
Nov 01 23:13:55.473751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473723    4531 cpu_manager.go:414] "containerMap: removing container" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main"
Nov 01 23:13:55.473818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473753    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main"
Nov 01 23:13:55.473818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473775    4531 cpu_manager.go:414] "containerMap: removing container" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main"
Nov 01 23:13:55.473818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473780    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main"
Nov 01 23:13:55.473818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473803    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e4921d3a-56d8-40db-a7ee-5039b886817b" containerName="revoke-keys"
Nov 01 23:13:55.473818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473809    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4921d3a-56d8-40db-a7ee-5039b886817b" containerName="revoke-keys"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473824    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerName="payload-pre-processing"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473829    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerName="payload-pre-processing"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473845    4531 cpu_manager.go:414] "containerMap: removing container" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerName="payload-processing"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473850    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerName="payload-processing"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473938    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f40cac-b1dc-407e-baed-3256eb2f0699" containerName="main"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473950    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ebfba5-06df-4469-8a21-8fd8f562dcea" containerName="main"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473957    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4921d3a-56d8-40db-a7ee-5039b886817b" containerName="revoke-keys"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473964    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b7bb7df-9c18-4e39-aa88-606a4da1a139" containerName="payload-pre-processing"
Nov 01 23:13:55.473979 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.473972    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="52362a8f-e57d-41dc-805c-bcfd024f143d" containerName="payload-processing"
Nov 01 23:13:55.478403 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.478378    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.484212 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.484185    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbfgq\" (UniqueName: \"kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.484334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.484302    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.484334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.484323    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.532520 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.532490    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"]
Nov 01 23:13:55.574474 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.574448    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:13:55.574618 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.574549    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:13:55.585119 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.585092    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.585254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.585133    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.585254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.585227    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbfgq\" (UniqueName: \"kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.585955 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.585935    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.587798 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.587779    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.640027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.640001    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbfgq\" (UniqueName: \"kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq\") pod \"maas-api-revoke-sub-175b4ea4-vsncm\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:55.791797 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:55.791701    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:56.172446 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:56.172411    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"]
Nov 01 23:13:56.172737 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:13:56.172699    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb641e3ae_e05c_4758_997d_3fee0d8f6828.slice/crio-86c72d266a406b36d2cf08b72bccb83836b98cf3fd2a15cadd55461f5fe64d29 WatchSource:0}: Error finding container 86c72d266a406b36d2cf08b72bccb83836b98cf3fd2a15cadd55461f5fe64d29: Status 404 returned error can't find the container with id 86c72d266a406b36d2cf08b72bccb83836b98cf3fd2a15cadd55461f5fe64d29
Nov 01 23:13:56.270220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:56.270189    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm" event={"ID":"b641e3ae-e05c-4758-997d-3fee0d8f6828","Type":"ContainerStarted","Data":"86c72d266a406b36d2cf08b72bccb83836b98cf3fd2a15cadd55461f5fe64d29"}
Nov 01 23:13:56.481873 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:56.481833    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:13:57.281371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:57.281283    4531 generic.go:356] "Generic (PLEG): container finished" podID="b641e3ae-e05c-4758-997d-3fee0d8f6828" containerID="4129ed1059ea9f343fce1b2e67759980983c032cc6f7c9e0336f5e9f388f9ef4" exitCode=0
Nov 01 23:13:57.281574 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:57.281369    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm" event={"ID":"b641e3ae-e05c-4758-997d-3fee0d8f6828","Type":"ContainerDied","Data":"4129ed1059ea9f343fce1b2e67759980983c032cc6f7c9e0336f5e9f388f9ef4"}
Nov 01 23:13:58.430462 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.430439    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:58.510529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.510498    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca\") pod \"b641e3ae-e05c-4758-997d-3fee0d8f6828\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") "
Nov 01 23:13:58.510673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.510543    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq\" (UniqueName: \"kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq\") pod \"b641e3ae-e05c-4758-997d-3fee0d8f6828\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") "
Nov 01 23:13:58.510673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.510601    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca\") pod \"b641e3ae-e05c-4758-997d-3fee0d8f6828\" (UID: \"b641e3ae-e05c-4758-997d-3fee0d8f6828\") "
Nov 01 23:13:58.510878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.510852    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca" pod "b641e3ae-e05c-4758-997d-3fee0d8f6828" (UID: "b641e3ae-e05c-4758-997d-3fee0d8f6828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:13:58.511016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.510998    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:58.512696 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.512672    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca" pod "b641e3ae-e05c-4758-997d-3fee0d8f6828" (UID: "b641e3ae-e05c-4758-997d-3fee0d8f6828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:13:58.512821 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.512714    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq" pod "b641e3ae-e05c-4758-997d-3fee0d8f6828" (UID: "b641e3ae-e05c-4758-997d-3fee0d8f6828"). InnerVolumeSpecName "kube-api-access-rbfgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:13:58.612180 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.612117    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbfgq\" (UniqueName: \"kubernetes.io/projected/b641e3ae-e05c-4758-997d-3fee0d8f6828-kube-api-access-rbfgq\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:58.612180 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:58.612142    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b641e3ae-e05c-4758-997d-3fee0d8f6828-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:13:59.296834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:59.296800    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"
Nov 01 23:13:59.297017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:59.296798    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm" event={"ID":"b641e3ae-e05c-4758-997d-3fee0d8f6828","Type":"ContainerDied","Data":"86c72d266a406b36d2cf08b72bccb83836b98cf3fd2a15cadd55461f5fe64d29"}
Nov 01 23:13:59.297017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:59.296949    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c72d266a406b36d2cf08b72bccb83836b98cf3fd2a15cadd55461f5fe64d29"
Nov 01 23:13:59.409621 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:59.409582    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:13:59.409621 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:13:59.409597    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:14:00.600075 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:00.600043    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:00.791221 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:00.791181    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:01.568222 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:01.568191    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:01.764682 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:01.764647    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:03.596326 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.596270    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 23:14:03.666239 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.666172    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume\") pod \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") "
Nov 01 23:14:03.666378 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.666311    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4\" (UniqueName: \"kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4\") pod \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\" (UID: \"a9f17394-841a-4aa3-ae19-88c12f1aaa2f\") "
Nov 01 23:14:03.666623 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.666594    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume" pod "a9f17394-841a-4aa3-ae19-88c12f1aaa2f" (UID: "a9f17394-841a-4aa3-ae19-88c12f1aaa2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:03.668421 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.668398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4" pod "a9f17394-841a-4aa3-ae19-88c12f1aaa2f" (UID: "a9f17394-841a-4aa3-ae19-88c12f1aaa2f"). InnerVolumeSpecName "kube-api-access-gwvt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:03.683794 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.683774    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 23:14:03.767052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.767021    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume\") pod \"f789004b-4add-4e83-bc23-f81ff62a7b25\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") "
Nov 01 23:14:03.767228 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.767166    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q\" (UniqueName: \"kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q\") pod \"f789004b-4add-4e83-bc23-f81ff62a7b25\" (UID: \"f789004b-4add-4e83-bc23-f81ff62a7b25\") "
Nov 01 23:14:03.767395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.767383    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwvt4\" (UniqueName: \"kubernetes.io/projected/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-kube-api-access-gwvt4\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:03.767435 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.767396    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9f17394-841a-4aa3-ae19-88c12f1aaa2f-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:03.767470 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.767438    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume" pod "f789004b-4add-4e83-bc23-f81ff62a7b25" (UID: "f789004b-4add-4e83-bc23-f81ff62a7b25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:03.769278 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.769256    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q" pod "f789004b-4add-4e83-bc23-f81ff62a7b25" (UID: "f789004b-4add-4e83-bc23-f81ff62a7b25"). InnerVolumeSpecName "kube-api-access-4q78q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:03.868211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.868134    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f789004b-4add-4e83-bc23-f81ff62a7b25-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:03.868211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:03.868162    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q78q\" (UniqueName: \"kubernetes.io/projected/f789004b-4add-4e83-bc23-f81ff62a7b25-kube-api-access-4q78q\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:04.336824 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.336781    4531 generic.go:356] "Generic (PLEG): container finished" podID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerID="c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103" exitCode=0
Nov 01 23:14:04.337037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.336853    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"
Nov 01 23:14:04.337037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.336873    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" event={"ID":"f789004b-4add-4e83-bc23-f81ff62a7b25","Type":"ContainerDied","Data":"c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103"}
Nov 01 23:14:04.337037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.336971    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr" event={"ID":"f789004b-4add-4e83-bc23-f81ff62a7b25","Type":"ContainerDied","Data":"7958a66ebded33a1a96eeb380c6d6c294cd1c6d5af31dcd42cae324a9067d658"}
Nov 01 23:14:04.337037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.337000    4531 scope.go:122] "RemoveContainer" containerID="c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103"
Nov 01 23:14:04.338554 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.338458    4531 generic.go:356] "Generic (PLEG): container finished" podID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerID="2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547" exitCode=0
Nov 01 23:14:04.338554 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.338527    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"
Nov 01 23:14:04.338687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.338581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" event={"ID":"a9f17394-841a-4aa3-ae19-88c12f1aaa2f","Type":"ContainerDied","Data":"2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547"}
Nov 01 23:14:04.338687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.338629    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj" event={"ID":"a9f17394-841a-4aa3-ae19-88c12f1aaa2f","Type":"ContainerDied","Data":"1381be3e455a8796900408e10853f06652fd5d23e4b1dd924e13e6f6af449b94"}
Nov 01 23:14:04.347520 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.347502    4531 scope.go:122] "RemoveContainer" containerID="c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103"
Nov 01 23:14:04.347818 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:04.347799    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103\": container with ID starting with c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103 not found: ID does not exist" containerID="c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103"
Nov 01 23:14:04.347898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.347824    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103"} err="failed to get container status \"c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103\": rpc error: code = NotFound desc = could not find container \"c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103\": container with ID starting with c5b4ecd17be452e9db3af32cab304e6b8c44cef7fa01d7437408579acde15103 not found: ID does not exist"
Nov 01 23:14:04.347898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.347841    4531 scope.go:122] "RemoveContainer" containerID="2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547"
Nov 01 23:14:04.360930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.360912    4531 scope.go:122] "RemoveContainer" containerID="2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547"
Nov 01 23:14:04.361196 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:04.361174    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547\": container with ID starting with 2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547 not found: ID does not exist" containerID="2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547"
Nov 01 23:14:04.361269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.361202    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547"} err="failed to get container status \"2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547\": rpc error: code = NotFound desc = could not find container \"2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547\": container with ID starting with 2e7bcfc711400404433883bc2c57da2778edd2ff850c348f8897f98d2d70a547 not found: ID does not exist"
Nov 01 23:14:04.494017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.493981    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"]
Nov 01 23:14:04.540426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.540395    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c768f3-5cc8c45b7-78ncr"]
Nov 01 23:14:04.641942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.641855    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"]
Nov 01 23:14:04.645787 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.645761    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c768f3-74f678b8df-cffkj"]
Nov 01 23:14:04.815481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.815451    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" path="/var/lib/kubelet/pods/a9f17394-841a-4aa3-ae19-88c12f1aaa2f/volumes"
Nov 01 23:14:04.815860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:04.815844    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" path="/var/lib/kubelet/pods/f789004b-4add-4e83-bc23-f81ff62a7b25/volumes"
Nov 01 23:14:05.305941 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:05.305900    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:05.485104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:05.485074    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:06.064952 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:06.064704    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:07.200329 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:07.200297    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:07.917826 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:07.917793    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:08.117266 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:08.117234    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:08.712820 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:08.712790    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:08.966289 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:08.966215    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:09.186410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:09.186380    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:09.625664 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:09.625633    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:09.810969 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:09.810935    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:10.048032 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:10.047998    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:10.649880 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:10.649846    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:12.526122 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:12.526083    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:14:14.165533 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:14.165490    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:15.285041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:15.285006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:15.468140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:15.468104    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:15.583122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:15.583032    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"]
Nov 01 23:14:15.583285 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:15.583241    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy" containerID="cri-o://96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a" gracePeriod=30
Nov 01 23:14:15.663790 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:15.663720    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:16.948471 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:16.948424    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:17.141129 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:17.141102    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:17.325057 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:17.324967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:17.732777 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:17.732744    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:19.195303 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:19.195267    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:20.841845 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.841820    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 23:14:20.942524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942461    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942504    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942614    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942642    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942687    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942715    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.942782 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942749    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:20.943140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942917    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:20.943140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942994    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.943140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.942997    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:20.943140 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943060    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8\") pod \"ddb49781-84b7-4132-9072-d40a0d545345\" (UID: \"ddb49781-84b7-4132-9072-d40a0d545345\") "
Nov 01 23:14:20.943359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943061    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:20.943359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943244    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:20.943586 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943563    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:20.943656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943592    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:20.943656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943609    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddb49781-84b7-4132-9072-d40a0d545345-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:20.943656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943623    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:20.943656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.943639    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:20.945223 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.945190    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:20.945422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.945398    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:14:20.945422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.945414    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:20.945602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:20.945455    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8" pod "ddb49781-84b7-4132-9072-d40a0d545345" (UID: "ddb49781-84b7-4132-9072-d40a0d545345"). InnerVolumeSpecName "kube-api-access-zmsx8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:21.045195 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.045152    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddb49781-84b7-4132-9072-d40a0d545345-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:21.045195 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.045195    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddb49781-84b7-4132-9072-d40a0d545345-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:21.045537 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.045214    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:21.045537 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.045226    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsx8\" (UniqueName: \"kubernetes.io/projected/ddb49781-84b7-4132-9072-d40a0d545345-kube-api-access-zmsx8\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:21.479178 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.479140    4531 generic.go:356] "Generic (PLEG): container finished" podID="ddb49781-84b7-4132-9072-d40a0d545345" containerID="96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a" exitCode=0
Nov 01 23:14:21.479376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.479202    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" event={"ID":"ddb49781-84b7-4132-9072-d40a0d545345","Type":"ContainerDied","Data":"96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a"}
Nov 01 23:14:21.479376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.479226    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"
Nov 01 23:14:21.479376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.479289    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv" event={"ID":"ddb49781-84b7-4132-9072-d40a0d545345","Type":"ContainerDied","Data":"776e64585e88ddcefc94835dd6c461ae3ca459ee2f2e543f09ad28d04c8501ee"}
Nov 01 23:14:21.479376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.479322    4531 scope.go:122] "RemoveContainer" containerID="96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a"
Nov 01 23:14:21.497284 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.497257    4531 scope.go:122] "RemoveContainer" containerID="96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a"
Nov 01 23:14:21.497609 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:21.497582    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a\": container with ID starting with 96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a not found: ID does not exist" containerID="96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a"
Nov 01 23:14:21.497696 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.497620    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a"} err="failed to get container status \"96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a\": rpc error: code = NotFound desc = could not find container \"96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a\": container with ID starting with 96efa0bda9b4efa7af3bfb9eb4233ed1d08b53d9aaa7a6eb3627a97fef65648a not found: ID does not exist"
Nov 01 23:14:21.633815 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.633782    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"]
Nov 01 23:14:21.680121 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:21.680084    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c768f3-openshift-default-78fc456855-2cxfv"]
Nov 01 23:14:22.643823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:22.643791    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"]
Nov 01 23:14:22.689925 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:22.689893    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"]
Nov 01 23:14:22.816486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:22.816455    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb49781-84b7-4132-9072-d40a0d545345" path="/var/lib/kubelet/pods/ddb49781-84b7-4132-9072-d40a0d545345/volumes"
Nov 01 23:14:22.850354 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:22.850329    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:24.411769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:24.411714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:29.231568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:29.231532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:29.669611 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:29.669541    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"]
Nov 01 23:14:29.771518 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:29.771489    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"]
Nov 01 23:14:30.082177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:30.082137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:30.264510 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:30.264474    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:33.933596 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:33.933554    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:34.670017 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:34.669981    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerName="payload-processing" containerID="cri-o://53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f" gracePeriod=25
Nov 01 23:14:34.771883 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:34.771850    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerName="payload-pre-processing" containerID="cri-o://338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee" gracePeriod=25
Nov 01 23:14:37.664217 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.664171    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" containerID="cri-o://2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164" gracePeriod=45
Nov 01 23:14:37.709019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.708964    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" containerID="cri-o://ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91" gracePeriod=45
Nov 01 23:14:37.876671 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.876332    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/ready\": dial tcp 10.133.0.75:8000: connect: connection refused"
Nov 01 23:14:37.956262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.956240    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 23:14:37.987821 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.987798    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 23:14:37.989899 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.989877    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm\") pod \"00109ff1-30af-470b-a0b1-a1b47d19ac07\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") "
Nov 01 23:14:37.989960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.989942    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx\" (UniqueName: \"kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx\") pod \"00109ff1-30af-470b-a0b1-a1b47d19ac07\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") "
Nov 01 23:14:37.989998 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.989990    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home\") pod \"00109ff1-30af-470b-a0b1-a1b47d19ac07\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") "
Nov 01 23:14:37.990044 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990010    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache\") pod \"00109ff1-30af-470b-a0b1-a1b47d19ac07\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") "
Nov 01 23:14:37.990117 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990092    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs\" (UniqueName: \"kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs\") pod \"00109ff1-30af-470b-a0b1-a1b47d19ac07\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") "
Nov 01 23:14:37.990253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990130    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir\") pod \"00109ff1-30af-470b-a0b1-a1b47d19ac07\" (UID: \"00109ff1-30af-470b-a0b1-a1b47d19ac07\") "
Nov 01 23:14:37.990253 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990196    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home" pod "00109ff1-30af-470b-a0b1-a1b47d19ac07" (UID: "00109ff1-30af-470b-a0b1-a1b47d19ac07"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:37.990369 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990294    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache" pod "00109ff1-30af-470b-a0b1-a1b47d19ac07" (UID: "00109ff1-30af-470b-a0b1-a1b47d19ac07"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:37.990458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990435    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir" pod "00109ff1-30af-470b-a0b1-a1b47d19ac07" (UID: "00109ff1-30af-470b-a0b1-a1b47d19ac07"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:37.990567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990552    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:37.990634 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990574    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:37.990634 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.990588    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:37.992113 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.992094    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx" pod "00109ff1-30af-470b-a0b1-a1b47d19ac07" (UID: "00109ff1-30af-470b-a0b1-a1b47d19ac07"). InnerVolumeSpecName "kube-api-access-2m4gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:37.992172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.992109    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm" pod "00109ff1-30af-470b-a0b1-a1b47d19ac07" (UID: "00109ff1-30af-470b-a0b1-a1b47d19ac07"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:37.992172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:37.992116    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs" pod "00109ff1-30af-470b-a0b1-a1b47d19ac07" (UID: "00109ff1-30af-470b-a0b1-a1b47d19ac07"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.091632    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm\") pod \"aafc9355-289e-42be-ba80-2514305c26de\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") "
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.091675    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache\") pod \"aafc9355-289e-42be-ba80-2514305c26de\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") "
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.091788    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home\") pod \"aafc9355-289e-42be-ba80-2514305c26de\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") "
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.091901    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir\") pod \"aafc9355-289e-42be-ba80-2514305c26de\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") "
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.091997    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache" pod "aafc9355-289e-42be-ba80-2514305c26de" (UID: "aafc9355-289e-42be-ba80-2514305c26de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092034    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs\" (UniqueName: \"kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs\") pod \"aafc9355-289e-42be-ba80-2514305c26de\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") "
Nov 01 23:14:38.092211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092162    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2\" (UniqueName: \"kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2\") pod \"aafc9355-289e-42be-ba80-2514305c26de\" (UID: \"aafc9355-289e-42be-ba80-2514305c26de\") "
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092219    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home" pod "aafc9355-289e-42be-ba80-2514305c26de" (UID: "aafc9355-289e-42be-ba80-2514305c26de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092428    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir" pod "aafc9355-289e-42be-ba80-2514305c26de" (UID: "aafc9355-289e-42be-ba80-2514305c26de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092492    4531 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-model-cache\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092513    4531 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-home\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092528    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00109ff1-30af-470b-a0b1-a1b47d19ac07-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092543    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00109ff1-30af-470b-a0b1-a1b47d19ac07-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.092739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.092556    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m4gx\" (UniqueName: \"kubernetes.io/projected/00109ff1-30af-470b-a0b1-a1b47d19ac07-kube-api-access-2m4gx\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.094425 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.094396    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs" pod "aafc9355-289e-42be-ba80-2514305c26de" (UID: "aafc9355-289e-42be-ba80-2514305c26de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:38.094569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.094556    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2" pod "aafc9355-289e-42be-ba80-2514305c26de" (UID: "aafc9355-289e-42be-ba80-2514305c26de"). InnerVolumeSpecName "kube-api-access-v9cw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:38.094894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.094878    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm" pod "aafc9355-289e-42be-ba80-2514305c26de" (UID: "aafc9355-289e-42be-ba80-2514305c26de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:14:38.120431 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.120400    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"]
Nov 01 23:14:38.121247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121225    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerName="payload-processing"
Nov 01 23:14:38.121301 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121252    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerName="payload-processing"
Nov 01 23:14:38.121301 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121297    4531 cpu_manager.go:414] "containerMap: removing container" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121306    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121318    4531 cpu_manager.go:414] "containerMap: removing container" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121326    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121337    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerName="payload-pre-processing"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121347    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerName="payload-pre-processing"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121360    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b641e3ae-e05c-4758-997d-3fee0d8f6828" containerName="revoke-keys"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121368    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b641e3ae-e05c-4758-997d-3fee0d8f6828" containerName="revoke-keys"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121394    4531 cpu_manager.go:414] "containerMap: removing container" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main"
Nov 01 23:14:38.121410 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121401    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main"
Nov 01 23:14:38.121688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121528    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f17394-841a-4aa3-ae19-88c12f1aaa2f" containerName="payload-processing"
Nov 01 23:14:38.121688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121551    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafc9355-289e-42be-ba80-2514305c26de" containerName="main"
Nov 01 23:14:38.121688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121563    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb49781-84b7-4132-9072-d40a0d545345" containerName="istio-proxy"
Nov 01 23:14:38.121688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121573    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f789004b-4add-4e83-bc23-f81ff62a7b25" containerName="payload-pre-processing"
Nov 01 23:14:38.121688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121584    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b641e3ae-e05c-4758-997d-3fee0d8f6828" containerName="revoke-keys"
Nov 01 23:14:38.121688 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.121595    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main"
Nov 01 23:14:38.126648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.126625    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.178303 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.178228    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"]
Nov 01 23:14:38.193406 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193373    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.193602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193436    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbh2\" (UniqueName: \"kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.193602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193524    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.193771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193650    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cw2\" (UniqueName: \"kubernetes.io/projected/aafc9355-289e-42be-ba80-2514305c26de-kube-api-access-v9cw2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.193771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193668    4531 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-dshm\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.193771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193681    4531 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aafc9355-289e-42be-ba80-2514305c26de-tmp-dir\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.193771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.193691    4531 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aafc9355-289e-42be-ba80-2514305c26de-tls-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:38.215434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.215402    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:14:38.215574 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.215487    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:14:38.294593 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.294564    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.294780 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.294606    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbh2\" (UniqueName: \"kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.294780 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.294628    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.295265 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.295243    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.296914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.296892    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.349240 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.349213    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbh2\" (UniqueName: \"kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2\") pod \"maas-api-revoke-sub-930272ce-vcc4d\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.446031 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.446012    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:38.594125 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.594095    4531 generic.go:356] "Generic (PLEG): container finished" podID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerID="2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164" exitCode=0
Nov 01 23:14:38.594286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.594146    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"
Nov 01 23:14:38.594286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.594190    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" event={"ID":"00109ff1-30af-470b-a0b1-a1b47d19ac07","Type":"ContainerDied","Data":"2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164"}
Nov 01 23:14:38.594286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.594233    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" event={"ID":"00109ff1-30af-470b-a0b1-a1b47d19ac07","Type":"ContainerDied","Data":"878378f1c8ba9663d62281acfc53dae7abeee7ae8720d87fa4dc059a3f1250a2"}
Nov 01 23:14:38.594286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.594249    4531 scope.go:122] "RemoveContainer" containerID="2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164"
Nov 01 23:14:38.595756 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.595718    4531 generic.go:356] "Generic (PLEG): container finished" podID="aafc9355-289e-42be-ba80-2514305c26de" containerID="ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91" exitCode=0
Nov 01 23:14:38.595889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.595760    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" event={"ID":"aafc9355-289e-42be-ba80-2514305c26de","Type":"ContainerDied","Data":"ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91"}
Nov 01 23:14:38.595889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.595780    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"
Nov 01 23:14:38.595889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.595793    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl" event={"ID":"aafc9355-289e-42be-ba80-2514305c26de","Type":"ContainerDied","Data":"892562fd7ec569a019f1f4865fb462f105764f0b6514bcd0bfdf83f07b915936"}
Nov 01 23:14:38.616469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.616445    4531 scope.go:122] "RemoveContainer" containerID="2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164"
Nov 01 23:14:38.616825 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:38.616796    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164\": container with ID starting with 2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164 not found: ID does not exist" containerID="2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164"
Nov 01 23:14:38.616938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.616835    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164"} err="failed to get container status \"2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164\": rpc error: code = NotFound desc = could not find container \"2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164\": container with ID starting with 2befa754c82cdd8aa426a0af4570b5586c0440fd31ff2495c6f8dc71fa428164 not found: ID does not exist"
Nov 01 23:14:38.616938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.616860    4531 scope.go:122] "RemoveContainer" containerID="ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91"
Nov 01 23:14:38.628810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.628793    4531 scope.go:122] "RemoveContainer" containerID="ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91"
Nov 01 23:14:38.629123 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:38.629085    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91\": container with ID starting with ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91 not found: ID does not exist" containerID="ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91"
Nov 01 23:14:38.629241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.629138    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91"} err="failed to get container status \"ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91\": rpc error: code = NotFound desc = could not find container \"ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91\": container with ID starting with ab6fac0b9c9e87fe5bee5e9b21952fcde20b4d45711668a8b0ad21ec63cb7e91 not found: ID does not exist"
Nov 01 23:14:38.748281 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.748220    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"]
Nov 01 23:14:38.752864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.752842    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/simulator-w4-9dfe9f-kserve-68d98d6698-s5sbl"]
Nov 01 23:14:38.815834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.815811    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aafc9355-289e-42be-ba80-2514305c26de" path="/var/lib/kubelet/pods/aafc9355-289e-42be-ba80-2514305c26de/volumes"
Nov 01 23:14:38.854153 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.854133    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"]
Nov 01 23:14:38.858663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.858642    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n"]
Nov 01 23:14:38.861527 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.861508    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"]
Nov 01 23:14:38.867283 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:14:38.867260    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97aeefec_eb5c_41cd_bc4f_c52c981d70fd.slice/crio-b8f818a35e5a5a6616f6f91bc055b8c61a9bea46486bf6d1ba5df32a678468d6 WatchSource:0}: Error finding container b8f818a35e5a5a6616f6f91bc055b8c61a9bea46486bf6d1ba5df32a678468d6: Status 404 returned error can't find the container with id b8f818a35e5a5a6616f6f91bc055b8c61a9bea46486bf6d1ba5df32a678468d6
Nov 01 23:14:38.888827 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:38.888797    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9dfe9f/distinct-2-w4-9dfe9f-kserve-6546fc75c7-tzv9n" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" containerName="main" probeResult="failure" output="Get \"https://10.133.0.105:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:14:39.606130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:39.606092    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d" event={"ID":"97aeefec-eb5c-41cd-bc4f-c52c981d70fd","Type":"ContainerStarted","Data":"7808fcd680c6539709aae7dd3f6a24cd0e56e2f7e6e61974ed00ca921da8e68f"}
Nov 01 23:14:39.606300 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:39.606151    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d" event={"ID":"97aeefec-eb5c-41cd-bc4f-c52c981d70fd","Type":"ContainerStarted","Data":"b8f818a35e5a5a6616f6f91bc055b8c61a9bea46486bf6d1ba5df32a678468d6"}
Nov 01 23:14:39.713538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:39.713489    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d" podStartSLOduration=1.713475377 podStartE2EDuration="1.713475377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:14:39.71198144 +0000 UTC m=+3061.029631454" watchObservedRunningTime="2026-11-01 23:14:39.713475377 +0000 UTC m=+3061.031125377"
Nov 01 23:14:40.614687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:40.614648    4531 generic.go:356] "Generic (PLEG): container finished" podID="97aeefec-eb5c-41cd-bc4f-c52c981d70fd" containerID="7808fcd680c6539709aae7dd3f6a24cd0e56e2f7e6e61974ed00ca921da8e68f" exitCode=0
Nov 01 23:14:40.615100 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:40.614693    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d" event={"ID":"97aeefec-eb5c-41cd-bc4f-c52c981d70fd","Type":"ContainerDied","Data":"7808fcd680c6539709aae7dd3f6a24cd0e56e2f7e6e61974ed00ca921da8e68f"}
Nov 01 23:14:40.815883 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:40.815852    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00109ff1-30af-470b-a0b1-a1b47d19ac07" path="/var/lib/kubelet/pods/00109ff1-30af-470b-a0b1-a1b47d19ac07/volumes"
Nov 01 23:14:41.048870 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.048841    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"]
Nov 01 23:14:41.051674 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.051646    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-782ff25b-4tr4m"]
Nov 01 23:14:41.772287 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.772262    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:41.828059 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.828031    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2\" (UniqueName: \"kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2\") pod \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") "
Nov 01 23:14:41.828220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.828083    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca\") pod \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") "
Nov 01 23:14:41.828220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.828151    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca\") pod \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\" (UID: \"97aeefec-eb5c-41cd-bc4f-c52c981d70fd\") "
Nov 01 23:14:41.828632 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.828590    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca" pod "97aeefec-eb5c-41cd-bc4f-c52c981d70fd" (UID: "97aeefec-eb5c-41cd-bc4f-c52c981d70fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:41.830181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.830152    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca" pod "97aeefec-eb5c-41cd-bc4f-c52c981d70fd" (UID: "97aeefec-eb5c-41cd-bc4f-c52c981d70fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:14:41.830275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.830207    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2" pod "97aeefec-eb5c-41cd-bc4f-c52c981d70fd" (UID: "97aeefec-eb5c-41cd-bc4f-c52c981d70fd"). InnerVolumeSpecName "kube-api-access-6bbh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:41.930353 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.930134    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbh2\" (UniqueName: \"kubernetes.io/projected/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-kube-api-access-6bbh2\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:41.930353 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.930166    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:41.930353 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:41.930183    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97aeefec-eb5c-41cd-bc4f-c52c981d70fd-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:42.526041 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:42.526004    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:14:42.630192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:42.630115    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"
Nov 01 23:14:42.630192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:42.630142    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d" event={"ID":"97aeefec-eb5c-41cd-bc4f-c52c981d70fd","Type":"ContainerDied","Data":"b8f818a35e5a5a6616f6f91bc055b8c61a9bea46486bf6d1ba5df32a678468d6"}
Nov 01 23:14:42.630192 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:42.630189    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8f818a35e5a5a6616f6f91bc055b8c61a9bea46486bf6d1ba5df32a678468d6"
Nov 01 23:14:42.816928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:42.816894    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a8ad203-051b-4a48-a659-3497e2c83883" path="/var/lib/kubelet/pods/4a8ad203-051b-4a48-a659-3497e2c83883/volumes"
Nov 01 23:14:44.442792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:44.442752    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:14:44.443204 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:44.442773    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:14:45.173701 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:45.173661    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:53.531660 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:53.531619    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:54.044779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:54.044745    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"]
Nov 01 23:14:54.091052 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:54.091024    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3ee9e5a-mv7vq"]
Nov 01 23:14:54.442581 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:54.442501    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:14:54.442581 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:54.442548    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 23:14:54.815882 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:54.815853    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea78909a-3497-4179-8f45-7c6221b15172" path="/var/lib/kubelet/pods/ea78909a-3497-4179-8f45-7c6221b15172/volumes"
Nov 01 23:14:54.922823 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:54.922786    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 23:14:55.021814 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.021792    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 23:14:55.022842 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.022819    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:55.048506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.048456    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume\") pod \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") "
Nov 01 23:14:55.048624 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.048582    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m\" (UniqueName: \"kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m\") pod \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\" (UID: \"79a8102a-eda2-4ae1-ab5d-a59b4555efcb\") "
Nov 01 23:14:55.048878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.048855    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume" pod "79a8102a-eda2-4ae1-ab5d-a59b4555efcb" (UID: "79a8102a-eda2-4ae1-ab5d-a59b4555efcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:55.050495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.050468    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m" pod "79a8102a-eda2-4ae1-ab5d-a59b4555efcb" (UID: "79a8102a-eda2-4ae1-ab5d-a59b4555efcb"). InnerVolumeSpecName "kube-api-access-xpq9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:55.149164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.149140    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume\") pod \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") "
Nov 01 23:14:55.149317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.149277    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k\" (UniqueName: \"kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k\") pod \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\" (UID: \"9fcb23f7-58a4-4c2b-bd70-d92a09214d42\") "
Nov 01 23:14:55.149487 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.149467    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume" pod "9fcb23f7-58a4-4c2b-bd70-d92a09214d42" (UID: "9fcb23f7-58a4-4c2b-bd70-d92a09214d42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:14:55.149555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.149538    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpq9m\" (UniqueName: \"kubernetes.io/projected/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-kube-api-access-xpq9m\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:55.149602 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.149561    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79a8102a-eda2-4ae1-ab5d-a59b4555efcb-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:55.151440 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.151413    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k" pod "9fcb23f7-58a4-4c2b-bd70-d92a09214d42" (UID: "9fcb23f7-58a4-4c2b-bd70-d92a09214d42"). InnerVolumeSpecName "kube-api-access-5h66k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:14:55.250704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.250679    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h66k\" (UniqueName: \"kubernetes.io/projected/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-kube-api-access-5h66k\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:55.250704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.250701    4531 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fcb23f7-58a4-4c2b-bd70-d92a09214d42-plugins-config-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:14:55.309290 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.309209    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:55.720104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.720072    4531 generic.go:356] "Generic (PLEG): container finished" podID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerID="53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f" exitCode=0
Nov 01 23:14:55.720271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.720122    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"
Nov 01 23:14:55.720271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.720160    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" event={"ID":"79a8102a-eda2-4ae1-ab5d-a59b4555efcb","Type":"ContainerDied","Data":"53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f"}
Nov 01 23:14:55.720271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.720223    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz" event={"ID":"79a8102a-eda2-4ae1-ab5d-a59b4555efcb","Type":"ContainerDied","Data":"4e1e786b8196ee7fb171731eb6b9cc11bc8f17505b67e21e8646b43fd919e1a1"}
Nov 01 23:14:55.720271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.720252    4531 scope.go:122] "RemoveContainer" containerID="53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f"
Nov 01 23:14:55.722657 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.722631    4531 generic.go:356] "Generic (PLEG): container finished" podID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerID="338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee" exitCode=0
Nov 01 23:14:55.722822 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.722794    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" event={"ID":"9fcb23f7-58a4-4c2b-bd70-d92a09214d42","Type":"ContainerDied","Data":"338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee"}
Nov 01 23:14:55.722894 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.722851    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c" event={"ID":"9fcb23f7-58a4-4c2b-bd70-d92a09214d42","Type":"ContainerDied","Data":"0719a8f4d5ebe1cdf9bca4ac87d5e689513a1ba36b7f5bccdadcfa78b3ce6ae2"}
Nov 01 23:14:55.722974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.722922    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"
Nov 01 23:14:55.732443 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.732428    4531 scope.go:122] "RemoveContainer" containerID="53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f"
Nov 01 23:14:55.732977 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:55.732820    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f\": container with ID starting with 53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f not found: ID does not exist" containerID="53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f"
Nov 01 23:14:55.732977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.732854    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f"} err="failed to get container status \"53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f\": rpc error: code = NotFound desc = could not find container \"53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f\": container with ID starting with 53958c092520ef971c3bdc9edb1fd44efc2649aa35591895f9851303e8558e6f not found: ID does not exist"
Nov 01 23:14:55.732977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.732876    4531 scope.go:122] "RemoveContainer" containerID="338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee"
Nov 01 23:14:55.743404 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.743379    4531 scope.go:122] "RemoveContainer" containerID="338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee"
Nov 01 23:14:55.743664 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:14:55.743644    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee\": container with ID starting with 338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee not found: ID does not exist" containerID="338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee"
Nov 01 23:14:55.743764 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.743674    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee"} err="failed to get container status \"338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee\": rpc error: code = NotFound desc = could not find container \"338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee\": container with ID starting with 338f26bfb329a7f34cef3a0b2ad06376192f00247278bb0f1d3a79081ccce0ee not found: ID does not exist"
Nov 01 23:14:55.881251 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.881221    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"]
Nov 01 23:14:55.926683 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.926649    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-9dfe9f-6f6d67dc9c-696hz"]
Nov 01 23:14:55.952809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.952785    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:55.989446 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.989419    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"]
Nov 01 23:14:55.995329 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:55.995303    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-9dfe9f-86945c59d7-9fd7c"]
Nov 01 23:14:56.814371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:56.814342    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" path="/var/lib/kubelet/pods/79a8102a-eda2-4ae1-ab5d-a59b4555efcb/volumes"
Nov 01 23:14:56.814703 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:56.814690    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" path="/var/lib/kubelet/pods/9fcb23f7-58a4-4c2b-bd70-d92a09214d42/volumes"
Nov 01 23:14:59.108434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:59.108386    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:14:59.300959 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:14:59.300924    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:01.121097 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:01.121061    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:02.334060 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:02.334022    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:02.515103 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:02.515076    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:02.628578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:02.628505    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"]
Nov 01 23:15:02.628748 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:02.628713    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy" containerID="cri-o://30c33a36db19f52ba58ce944bce8b129efc08ca3221fc27c54383f4e8497b788" gracePeriod=30
Nov 01 23:15:02.712156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:02.712126    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:06.672672 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:06.672638    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:07.807376 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.807346    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-5vq6r"]
Nov 01 23:15:07.808154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.808100    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager" containerID="cri-o://3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0" gracePeriod=30
Nov 01 23:15:07.809948 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.809920    4531 generic.go:356] "Generic (PLEG): container finished" podID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerID="30c33a36db19f52ba58ce944bce8b129efc08ca3221fc27c54383f4e8497b788" exitCode=0
Nov 01 23:15:07.810069 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.810002    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" event={"ID":"e9bd15c1-bd84-460b-b07a-0e77a6aed64e","Type":"ContainerDied","Data":"30c33a36db19f52ba58ce944bce8b129efc08ca3221fc27c54383f4e8497b788"}
Nov 01 23:15:07.881363 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.881340    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 23:15:07.968807 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.968778    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.968836    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.968859    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.968900    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.968963    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.968997    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969027    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969065    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:15:07.969313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969108    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969152    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:15:07.969313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969170    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token\") pod \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\" (UID: \"e9bd15c1-bd84-460b-b07a-0e77a6aed64e\") "
Nov 01 23:15:07.969598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969408    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:15:07.969598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969438    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:15:07.969598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969527    4531 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-credential-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:07.969598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969547    4531 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:07.969598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969561    4531 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istiod-ca-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:07.969598 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969576    4531 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-workload-socket\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:07.970014 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.969784    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:15:07.971346 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.971307    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 23:15:07.971493 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.971376    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:15:07.971572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.971510    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:15:07.971572 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:07.971534    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5" pod "e9bd15c1-bd84-460b-b07a-0e77a6aed64e" (UID: "e9bd15c1-bd84-460b-b07a-0e77a6aed64e"). InnerVolumeSpecName "kube-api-access-c98m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:15:08.051852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.051829    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 23:15:08.070833 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.070810    4531 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-envoy\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.070833 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.070830    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98m5\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-kube-api-access-c98m5\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.070967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.070841    4531 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-token\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.070967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.070849    4531 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-data\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.070967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.070857    4531 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e9bd15c1-bd84-460b-b07a-0e77a6aed64e-istio-podinfo\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.171513 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.171452    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs\") pod \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") "
Nov 01 23:15:08.171651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.171526    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert\") pod \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") "
Nov 01 23:15:08.171651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.171562    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq\" (UniqueName: \"kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq\") pod \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\" (UID: \"7fb92f36-91c2-4163-9ffb-c665e3059bd9\") "
Nov 01 23:15:08.173473 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.173447    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs" pod "7fb92f36-91c2-4163-9ffb-c665e3059bd9" (UID: "7fb92f36-91c2-4163-9ffb-c665e3059bd9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:15:08.173659 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.173631    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert" pod "7fb92f36-91c2-4163-9ffb-c665e3059bd9" (UID: "7fb92f36-91c2-4163-9ffb-c665e3059bd9"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:15:08.173879 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.173708    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq" pod "7fb92f36-91c2-4163-9ffb-c665e3059bd9" (UID: "7fb92f36-91c2-4163-9ffb-c665e3059bd9"). InnerVolumeSpecName "kube-api-access-ds2lq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:15:08.273188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.273159    4531 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-metrics-certs\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.273188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.273185    4531 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7fb92f36-91c2-4163-9ffb-c665e3059bd9-webhook-cert\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.273188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.273195    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds2lq\" (UniqueName: \"kubernetes.io/projected/7fb92f36-91c2-4163-9ffb-c665e3059bd9-kube-api-access-ds2lq\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:15:08.818411 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.818381    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"
Nov 01 23:15:08.818411 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.818391    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq" event={"ID":"e9bd15c1-bd84-460b-b07a-0e77a6aed64e","Type":"ContainerDied","Data":"4dd585a1dbd907665a480aee9bf59c7bf1e5295eaeaf5e3e8ed2adfbcc468ac5"}
Nov 01 23:15:08.819009 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.818464    4531 scope.go:122] "RemoveContainer" containerID="30c33a36db19f52ba58ce944bce8b129efc08ca3221fc27c54383f4e8497b788"
Nov 01 23:15:08.820662 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.820632    4531 generic.go:356] "Generic (PLEG): container finished" podID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerID="3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0" exitCode=0
Nov 01 23:15:08.820921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.820835    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" event={"ID":"7fb92f36-91c2-4163-9ffb-c665e3059bd9","Type":"ContainerDied","Data":"3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0"}
Nov 01 23:15:08.820921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.820889    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r" event={"ID":"7fb92f36-91c2-4163-9ffb-c665e3059bd9","Type":"ContainerDied","Data":"345b32f7f3ed4e7600a301b296afc09fba16fe6fda557c229ed40c984e4540bc"}
Nov 01 23:15:08.820921 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.820907    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5c9f79b66c-5vq6r"
Nov 01 23:15:08.831547 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.831530    4531 scope.go:122] "RemoveContainer" containerID="3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0"
Nov 01 23:15:08.841418 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.841399    4531 scope.go:122] "RemoveContainer" containerID="d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"
Nov 01 23:15:08.851297 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.851275    4531 scope.go:122] "RemoveContainer" containerID="3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0"
Nov 01 23:15:08.851571 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:15:08.851552    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0\": container with ID starting with 3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0 not found: ID does not exist" containerID="3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0"
Nov 01 23:15:08.851636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.851579    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0"} err="failed to get container status \"3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0\": rpc error: code = NotFound desc = could not find container \"3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0\": container with ID starting with 3103bfd1450fd7c5c3363b8d1c6a4c7419fa986b59925144e7fbfa326bbb0cd0 not found: ID does not exist"
Nov 01 23:15:08.851636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.851596    4531 scope.go:122] "RemoveContainer" containerID="d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"
Nov 01 23:15:08.851923 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:15:08.851888    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943\": container with ID starting with d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943 not found: ID does not exist" containerID="d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"
Nov 01 23:15:08.851971 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:08.851935    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943"} err="failed to get container status \"d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943\": rpc error: code = NotFound desc = could not find container \"d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943\": container with ID starting with d018cf05d28fb4c4e0c2c53d351413150d5dced75c47dbe360136ba84c58d943 not found: ID does not exist"
Nov 01 23:15:09.019957 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:09.019918    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"]
Nov 01 23:15:09.066109 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:09.066079    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9dfe9f-openshift-default-85c786884c-6h5mq"]
Nov 01 23:15:09.126856 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:09.126827    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-5vq6r"]
Nov 01 23:15:09.130752 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:09.130717    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5c9f79b66c-5vq6r"]
Nov 01 23:15:10.814739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:10.814687    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" path="/var/lib/kubelet/pods/7fb92f36-91c2-4163-9ffb-c665e3059bd9/volumes"
Nov 01 23:15:10.815375 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:10.815353    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" path="/var/lib/kubelet/pods/e9bd15c1-bd84-460b-b07a-0e77a6aed64e/volumes"
Nov 01 23:15:12.527318 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:15:12.527278    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:15:25.187813 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:25.187718    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:42.329142 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:42.329112    4531 scope.go:122] "RemoveContainer" containerID="b464b290a486bff61fc420d384a1e99a2d92d05f83da49616a1de598ef28f5cc"
Nov 01 23:15:42.338505 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:42.338487    4531 scope.go:122] "RemoveContainer" containerID="0243c85fa1ead2e2bf5c210f3d9c987e32a612c397c6677361baf0eb45c005c0"
Nov 01 23:15:42.524894 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:15:42.524860    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:15:45.431219 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:45.431183    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:46.443476 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:46.443442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:48.876541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:48.876505    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:15:58.043844 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:58.043808    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"]
Nov 01 23:15:58.090364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:58.090333    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89bf61fd-xs4pq"]
Nov 01 23:15:58.814772 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:15:58.814721    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e40b2f7-1082-453b-b708-8aa401beb6b2" path="/var/lib/kubelet/pods/1e40b2f7-1082-453b-b708-8aa401beb6b2/volumes"
Nov 01 23:16:12.524865 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:16:12.524827    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:16:18.046565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:16:18.046533    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"]
Nov 01 23:16:18.091093 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:16:18.091060    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad6086ab-75v8v"]
Nov 01 23:16:18.814617 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:16:18.814586    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89175acc-4741-45d4-976e-d018f1f09968" path="/var/lib/kubelet/pods/89175acc-4741-45d4-976e-d018f1f09968/volumes"
Nov 01 23:16:42.382299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:16:42.382269    4531 scope.go:122] "RemoveContainer" containerID="505e74e8c69b78d028f692af7d340b06f52111893e63707014c6d1fb2ddac496"
Nov 01 23:16:42.391483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:16:42.391459    4531 scope.go:122] "RemoveContainer" containerID="57ca563580bd99a3d8bb9cc99c157f0763cb6441a8fedf7865a57cbc0906cf4a"
Nov 01 23:16:42.524437 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:16:42.524404    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:16:47.743984 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:16:47.743950    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:17:06.046215 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:06.046136    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"]
Nov 01 23:17:06.093676 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:06.093604    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"]
Nov 01 23:17:06.096166 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:06.096142    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cb95de2-l2pmj"]
Nov 01 23:17:06.098429 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:06.098403    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b6983c3-lfg9g"]
Nov 01 23:17:06.814592 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:06.814550    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d73b77f-1b2d-41d6-9ddd-73022427d2df" path="/var/lib/kubelet/pods/4d73b77f-1b2d-41d6-9ddd-73022427d2df/volumes"
Nov 01 23:17:06.815103 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:06.815081    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55ae1d3-2743-4f98-b0d1-5711b4c0950a" path="/var/lib/kubelet/pods/a55ae1d3-2743-4f98-b0d1-5711b4c0950a/volumes"
Nov 01 23:17:10.530497 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.530463    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"]
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531431    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531457    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531492    4531 cpu_manager.go:414] "containerMap: removing container" podUID="97aeefec-eb5c-41cd-bc4f-c52c981d70fd" containerName="revoke-keys"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531500    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97aeefec-eb5c-41cd-bc4f-c52c981d70fd" containerName="revoke-keys"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531553    4531 cpu_manager.go:414] "containerMap: removing container" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerName="payload-pre-processing"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531562    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerName="payload-pre-processing"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531603    4531 cpu_manager.go:414] "containerMap: removing container" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerName="payload-processing"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531612    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerName="payload-processing"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531645    4531 cpu_manager.go:414] "containerMap: removing container" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy"
Nov 01 23:17:10.531641 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531653    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy"
Nov 01 23:17:10.532330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531869    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9bd15c1-bd84-460b-b07a-0e77a6aed64e" containerName="istio-proxy"
Nov 01 23:17:10.532330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531897    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager"
Nov 01 23:17:10.532330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531908    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="97aeefec-eb5c-41cd-bc4f-c52c981d70fd" containerName="revoke-keys"
Nov 01 23:17:10.532330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531931    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fcb23f7-58a4-4c2b-bd70-d92a09214d42" containerName="payload-pre-processing"
Nov 01 23:17:10.532330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.531950    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="79a8102a-eda2-4ae1-ab5d-a59b4555efcb" containerName="payload-processing"
Nov 01 23:17:10.536233 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.536204    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.690576 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690542    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.690749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690586    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.690749 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690675    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.690857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690776    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zv84\" (UniqueName: \"kubernetes.io/projected/da2a2592-ce65-4dcc-add0-1031bc2a1079-kube-api-access-4zv84\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.690857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690797    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2a2592-ce65-4dcc-add0-1031bc2a1079-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.690857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690823    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.691034 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.690987    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.691089 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.691060    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.691149 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.691101    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.724869 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.724841    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"]
Nov 01 23:17:10.792559 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792488    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.792559 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792525    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.792559 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792545    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.792834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792570    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.792834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792762    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.792834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792826    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zv84\" (UniqueName: \"kubernetes.io/projected/da2a2592-ce65-4dcc-add0-1031bc2a1079-kube-api-access-4zv84\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792849    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2a2592-ce65-4dcc-add0-1031bc2a1079-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792877    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792925    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.792954    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793041 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.793027    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.793180    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793333 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.793299    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.793618 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.793599    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da2a2592-ce65-4dcc-add0-1031bc2a1079-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.794814 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.794791    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.795233 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.795211    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.845477 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.845449    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da2a2592-ce65-4dcc-add0-1031bc2a1079-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.845618 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.845513    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zv84\" (UniqueName: \"kubernetes.io/projected/da2a2592-ce65-4dcc-add0-1031bc2a1079-kube-api-access-4zv84\") pod \"maas-default-gateway-openshift-default-7f694df8bc-glttk\" (UID: \"da2a2592-ce65-4dcc-add0-1031bc2a1079\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:10.851371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:10.851352    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:11.272191 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.272148    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"]
Nov 01 23:17:11.277472 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:17:11.277442    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda2a2592_ce65_4dcc_add0_1031bc2a1079.slice/crio-df0581372b737b595d02572a57b77559e545b5926bda40a514475fd7fa6307d0 WatchSource:0}: Error finding container df0581372b737b595d02572a57b77559e545b5926bda40a514475fd7fa6307d0: Status 404 returned error can't find the container with id df0581372b737b595d02572a57b77559e545b5926bda40a514475fd7fa6307d0
Nov 01 23:17:11.279502 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.279471    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:17:11.279603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.279537    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:17:11.279603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.279564    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:17:11.574900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.574828    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" event={"ID":"da2a2592-ce65-4dcc-add0-1031bc2a1079","Type":"ContainerStarted","Data":"775b80b84508ddb7678ffc4ea2dcb325a93308badf9aa1137a60e84686443727"}
Nov 01 23:17:11.574900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.574880    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" event={"ID":"da2a2592-ce65-4dcc-add0-1031bc2a1079","Type":"ContainerStarted","Data":"df0581372b737b595d02572a57b77559e545b5926bda40a514475fd7fa6307d0"}
Nov 01 23:17:11.686357 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.686307    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podStartSLOduration=1.686288261 podStartE2EDuration="1.686288261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:17:11.685275437 +0000 UTC m=+3213.002925474" watchObservedRunningTime="2026-11-01 23:17:11.686288261 +0000 UTC m=+3213.003938263"
Nov 01 23:17:11.851857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:11.851788    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:12.526097 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:17:12.526061    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:17:12.853247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:12.853173    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-glttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 01 23:17:12.853247 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:12.853226    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podUID="da2a2592-ce65-4dcc-add0-1031bc2a1079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 01 23:17:13.852031 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:13.851993    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-glttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 01 23:17:13.852246 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:13.852048    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podUID="da2a2592-ce65-4dcc-add0-1031bc2a1079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 01 23:17:14.851927 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:14.851894    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-glttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 01 23:17:14.852338 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:14.851944    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podUID="da2a2592-ce65-4dcc-add0-1031bc2a1079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 01 23:17:15.852552 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:15.852517    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-glttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 01 23:17:15.852967 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:15.852572    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podUID="da2a2592-ce65-4dcc-add0-1031bc2a1079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 01 23:17:16.852159 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:16.852127    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-glttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 01 23:17:16.852347 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:16.852180    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podUID="da2a2592-ce65-4dcc-add0-1031bc2a1079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 01 23:17:17.852434 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:17.852401    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-glttk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body=
Nov 01 23:17:17.852851 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:17.852448    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk" podUID="da2a2592-ce65-4dcc-add0-1031bc2a1079" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused"
Nov 01 23:17:19.458534 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:19.458499    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:19.459049 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:19.458847    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:19.459670 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:19.459641    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-glttk"
Nov 01 23:17:42.417906 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:42.417871    4531 scope.go:122] "RemoveContainer" containerID="7dacdaf4c31fe5344b13bb1a279d9987ae3a9ba1fac3d8f429c8d39d6dc3fef0"
Nov 01 23:17:42.427966 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:42.427944    4531 scope.go:122] "RemoveContainer" containerID="ff6e9449d43ad4d775d453af03f6dfb2658f1749a970ab925decb400d08b59aa"
Nov 01 23:17:42.436122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:42.436103    4531 scope.go:122] "RemoveContainer" containerID="af0c4e067e6d5a30750dbacfc5595c71cbcd931b53d1ad206f7c5b6516bf363a"
Nov 01 23:17:42.444288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:42.444271    4531 scope.go:122] "RemoveContainer" containerID="1d63789fe91f5b5f1cad7e4ecc098a41e7e99c809ace85c45c1190b38e3ce410"
Nov 01 23:17:42.525064 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:17:42.525027    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:17:46.084872 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:17:46.084828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:18:05.980145 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:05.980110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:18:11.000383 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.000352    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"]
Nov 01 23:18:11.000812 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.000789    4531 cpu_manager.go:414] "containerMap: removing container" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager"
Nov 01 23:18:11.000812 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.000803    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager"
Nov 01 23:18:11.000910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.000896    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb92f36-91c2-4163-9ffb-c665e3059bd9" containerName="manager"
Nov 01 23:18:11.004147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.004127    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120528 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120496    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120683 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120533    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120683 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120559    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cftjz\" (UniqueName: \"kubernetes.io/projected/31480339-9503-4930-bfb6-3d8c4b2680ca-kube-api-access-cftjz\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120789 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120676    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120789 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120763    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120864 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120843    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120903 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120888    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.120963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120949    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.121003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.120970    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31480339-9503-4930-bfb6-3d8c4b2680ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.196246 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.196216    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"]
Nov 01 23:18:11.221415 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221388    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221442    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221476    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221508    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221528    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31480339-9503-4930-bfb6-3d8c4b2680ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221565 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221551    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221806 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221573    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221806 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221594    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cftjz\" (UniqueName: \"kubernetes.io/projected/31480339-9503-4930-bfb6-3d8c4b2680ca-kube-api-access-cftjz\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221806 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221612    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221941 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221897    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-credential-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.221941 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.221910    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-data\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.222154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.222133    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-workload-certs\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.222348 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.222323    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-workload-socket\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.222610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.222586    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31480339-9503-4930-bfb6-3d8c4b2680ca-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.224164 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.224140    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-envoy\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.225244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.225219    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.275074 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.275007    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31480339-9503-4930-bfb6-3d8c4b2680ca-istio-token\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.275199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.275135    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cftjz\" (UniqueName: \"kubernetes.io/projected/31480339-9503-4930-bfb6-3d8c4b2680ca-kube-api-access-cftjz\") pod \"maas-default-gateway-openshift-default-7f694df8bc-4txmj\" (UID: \"31480339-9503-4930-bfb6-3d8c4b2680ca\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.317033 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.317010    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:11.736155 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.736125    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"]
Nov 01 23:18:11.741177 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:18:11.741152    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31480339_9503_4930_bfb6_3d8c4b2680ca.slice/crio-33c76ccde5becaac0696849fc9e4d057ea14a0aa8495c56ec0dc7abc3b12c591 WatchSource:0}: Error finding container 33c76ccde5becaac0696849fc9e4d057ea14a0aa8495c56ec0dc7abc3b12c591: Status 404 returned error can't find the container with id 33c76ccde5becaac0696849fc9e4d057ea14a0aa8495c56ec0dc7abc3b12c591
Nov 01 23:18:11.743244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.743214    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:18:11.743326 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.743279    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:18:11.743326 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.743309    4531 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Nov 01 23:18:11.983452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.983415    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" event={"ID":"31480339-9503-4930-bfb6-3d8c4b2680ca","Type":"ContainerStarted","Data":"8a2d5051557d15b38e7f9267f5cc3254b3a2eb0e415e07fff6ac01a7e8121cc4"}
Nov 01 23:18:11.983645 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:11.983482    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" event={"ID":"31480339-9503-4930-bfb6-3d8c4b2680ca","Type":"ContainerStarted","Data":"33c76ccde5becaac0696849fc9e4d057ea14a0aa8495c56ec0dc7abc3b12c591"}
Nov 01 23:18:12.097457 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:12.097360    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podStartSLOduration=2.097345742 podStartE2EDuration="2.097345742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:18:12.094050872 +0000 UTC m=+3273.411700874" watchObservedRunningTime="2026-11-01 23:18:12.097345742 +0000 UTC m=+3273.414995743"
Nov 01 23:18:12.317235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:12.317198    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:12.318879 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:12.318851    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:12.319042 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:12.318896    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:12.525846 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:18:12.525810    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:18:13.318468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:13.318429    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:13.319122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:13.318501    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:14.317436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:14.317402    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:14.317700 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:14.317666    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:15.317409 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:15.317374    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:15.317909 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:15.317426    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:16.318146 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:16.318106    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:16.318801 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:16.318769    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:17.318241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:17.318202    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:17.318886 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:17.318851    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:18.317840 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:18.317800    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:18.318030 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:18.317878    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:19.318148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:19.318116    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused" start-of-body=
Nov 01 23:18:19.318541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:19.318168    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": dial tcp 10.133.0.248:15021: connect: connection refused"
Nov 01 23:18:21.358867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:21.358828    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:18:21.359288 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:21.358932    4531 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:18:22.138283 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:22.138224    4531 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:22.138655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:22.138631    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:22.139391 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:22.139373    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj"
Nov 01 23:18:42.480760 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:42.480713    4531 scope.go:122] "RemoveContainer" containerID="7c242fc1e1ca6bf73dc1650122a61942dbf4e843e3e59bc71b895089f9b34125"
Nov 01 23:18:42.490023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:42.490007    4531 scope.go:122] "RemoveContainer" containerID="3bb4d78a4716957eab983a3d7e73681742c24d46cb1e0494b49e49e6fd375bff"
Nov 01 23:18:42.498230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:42.498212    4531 scope.go:122] "RemoveContainer" containerID="986330da053a4c0c6f23924ed6c0a66f289ec2330f9ea73b78cb244cba90756f"
Nov 01 23:18:42.506026 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:42.506007    4531 scope.go:122] "RemoveContainer" containerID="21fce33acb9fda746577af0b79cefad465776c2dc86fabbd1dcebf92c0eaccf1"
Nov 01 23:18:42.515161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:42.515136    4531 scope.go:122] "RemoveContainer" containerID="10e48b6dc278e8cbfe49e78ace030e22495dd8f632afa1f97d14209ed90323a4"
Nov 01 23:18:42.526535 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:18:42.526506    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:18:46.368671 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:46.368640    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:18:59.044539 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:59.044502    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"]
Nov 01 23:18:59.089789 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:18:59.089761    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-175b4ea4-vsncm"]
Nov 01 23:19:00.822830 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:00.822795    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b641e3ae-e05c-4758-997d-3fee0d8f6828" path="/var/lib/kubelet/pods/b641e3ae-e05c-4758-997d-3fee0d8f6828/volumes"
Nov 01 23:19:04.026436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:04.026367    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:19:04.259335 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:04.259304    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:19:10.133013 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:10.132982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:19:12.525449 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:19:12.525413    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:19:36.088614 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:36.088575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:19:42.044395 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:42.044362    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"]
Nov 01 23:19:42.095312 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:42.092022    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-930272ce-vcc4d"]
Nov 01 23:19:42.527002 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:19:42.526967    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:19:42.553715 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:42.553685    4531 scope.go:122] "RemoveContainer" containerID="c260e74c51240413922f0c4c87ce19acdfc673100907b26856f2f5e449a28e8d"
Nov 01 23:19:42.815458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:42.815389    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97aeefec-eb5c-41cd-bc4f-c52c981d70fd" path="/var/lib/kubelet/pods/97aeefec-eb5c-41cd-bc4f-c52c981d70fd/volumes"
Nov 01 23:19:46.756196 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:19:46.756164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:20:12.525691 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:20:12.525653    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:20:41.045448 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:20:41.045413    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:20:42.525741 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:20:42.525692    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:20:42.576570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:20:42.576539    4531 scope.go:122] "RemoveContainer" containerID="4129ed1059ea9f343fce1b2e67759980983c032cc6f7c9e0336f5e9f388f9ef4"
Nov 01 23:20:42.585438 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:20:42.585420    4531 scope.go:122] "RemoveContainer" containerID="7808fcd680c6539709aae7dd3f6a24cd0e56e2f7e6e61974ed00ca921da8e68f"
Nov 01 23:20:47.081571 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:20:47.081538    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:21:12.538148 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:21:12.538105    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:21:42.525831 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:21:42.525707    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:21:47.537736 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:21:47.537703    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:21:57.311170 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:21:57.311129    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:21:57.911199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:21:57.911164    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:21:58.217555 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:21:58.217522    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:01.782536 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:01.782500    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:01.978942 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:01.978908    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:02.162112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:02.162030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:05.053089 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:05.053056    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:05.251871 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:05.251835    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:08.079513 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:08.079479    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:08.312472 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:08.312435    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:08.535662 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:08.535634    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:12.525107 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:22:12.525070    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:22:12.991333 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:12.991279    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:17.892580 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:17.892544    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:22.714405 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:22.714364    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:26.553968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:26.553935    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:27.125276 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:27.125244    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:27.335212 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:27.335175    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:28.019299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:28.019218    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:30.360398 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:30.360354    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:32.801768 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:32.801721    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:37.702299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:37.702263    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:42.497658 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:42.497613    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:42.526103 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:22:42.526062    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:22:47.299947 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:47.299862    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:47.763955 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:47.763923    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:52.186986 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:52.186884    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:52.370527 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:52.370493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:22:56.995873 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:22:56.995828    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:01.797120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:01.797079    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:06.668964 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:06.668927    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:11.417414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:11.417355    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:12.526490 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:23:12.526441    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:23:16.141045 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:16.141011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:19.461815 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.461778    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"]
Nov 01 23:23:19.465786 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.465767    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.506860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.506837    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbn7w\" (UniqueName: \"kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.506974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.506910    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.507024 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.506993    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.561943 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.561915    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:23:19.562283 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.562263    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:23:19.565055 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.565031    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"]
Nov 01 23:23:19.607420 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.607395    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.607542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.607428    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.607542 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.607477    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbn7w\" (UniqueName: \"kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.608152 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.608134    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.609666 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.609648    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.663038 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.663010    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbn7w\" (UniqueName: \"kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w\") pod \"maas-api-revoke-sub-93a8cb84-vlxjt\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:19.779155 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:19.779102    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:20.149669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:20.149587    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"]
Nov 01 23:23:20.153835 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:23:20.153803    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a63558c_0214_4e85_a413_d13728cd5e3f.slice/crio-2ff9609fd05e431c1803b60393107a20b655e10abb20aae365909973cd985384 WatchSource:0}: Error finding container 2ff9609fd05e431c1803b60393107a20b655e10abb20aae365909973cd985384: Status 404 returned error can't find the container with id 2ff9609fd05e431c1803b60393107a20b655e10abb20aae365909973cd985384
Nov 01 23:23:20.899552 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:20.899471    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt" event={"ID":"2a63558c-0214-4e85-a413-d13728cd5e3f","Type":"ContainerStarted","Data":"3fecd1c68d9af883dc35da053b530f78cf53e401c35ce9ccbdf20e86f94ca1c9"}
Nov 01 23:23:20.899552 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:20.899519    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt" event={"ID":"2a63558c-0214-4e85-a413-d13728cd5e3f","Type":"ContainerStarted","Data":"2ff9609fd05e431c1803b60393107a20b655e10abb20aae365909973cd985384"}
Nov 01 23:23:21.011144 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:21.011092    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt" podStartSLOduration=2.011077428 podStartE2EDuration="2.011077428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:23:21.007244858 +0000 UTC m=+3582.324894859" watchObservedRunningTime="2026-11-01 23:23:21.011077428 +0000 UTC m=+3582.328727429"
Nov 01 23:23:21.016390 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:21.016353    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:21.906842 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:21.906805    4531 generic.go:356] "Generic (PLEG): container finished" podID="2a63558c-0214-4e85-a413-d13728cd5e3f" containerID="3fecd1c68d9af883dc35da053b530f78cf53e401c35ce9ccbdf20e86f94ca1c9" exitCode=0
Nov 01 23:23:21.907265 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:21.906899    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt" event={"ID":"2a63558c-0214-4e85-a413-d13728cd5e3f","Type":"ContainerDied","Data":"3fecd1c68d9af883dc35da053b530f78cf53e401c35ce9ccbdf20e86f94ca1c9"}
Nov 01 23:23:23.050686 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.050664    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:23.138457 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.138434    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca\") pod \"2a63558c-0214-4e85-a413-d13728cd5e3f\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") "
Nov 01 23:23:23.138585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.138539    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca\") pod \"2a63558c-0214-4e85-a413-d13728cd5e3f\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") "
Nov 01 23:23:23.138585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.138572    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w\" (UniqueName: \"kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w\") pod \"2a63558c-0214-4e85-a413-d13728cd5e3f\" (UID: \"2a63558c-0214-4e85-a413-d13728cd5e3f\") "
Nov 01 23:23:23.138812 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.138787    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca" pod "2a63558c-0214-4e85-a413-d13728cd5e3f" (UID: "2a63558c-0214-4e85-a413-d13728cd5e3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:23:23.140507 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.140480    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w" pod "2a63558c-0214-4e85-a413-d13728cd5e3f" (UID: "2a63558c-0214-4e85-a413-d13728cd5e3f"). InnerVolumeSpecName "kube-api-access-zbn7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:23:23.140642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.140621    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca" pod "2a63558c-0214-4e85-a413-d13728cd5e3f" (UID: "2a63558c-0214-4e85-a413-d13728cd5e3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:23:23.240093 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.240064    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbn7w\" (UniqueName: \"kubernetes.io/projected/2a63558c-0214-4e85-a413-d13728cd5e3f-kube-api-access-zbn7w\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:23.240093 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.240092    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:23.240264 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.240106    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a63558c-0214-4e85-a413-d13728cd5e3f-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:23.919852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.919822    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"
Nov 01 23:23:23.919852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.919840    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt" event={"ID":"2a63558c-0214-4e85-a413-d13728cd5e3f","Type":"ContainerDied","Data":"2ff9609fd05e431c1803b60393107a20b655e10abb20aae365909973cd985384"}
Nov 01 23:23:23.920115 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:23.919880    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff9609fd05e431c1803b60393107a20b655e10abb20aae365909973cd985384"
Nov 01 23:23:25.795131 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:25.795090    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:30.596006 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:30.595970    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:31.188068 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.188032    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:31.666788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.666755    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"]
Nov 01 23:23:31.667443 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.667420    4531 cpu_manager.go:414] "containerMap: removing container" podUID="2a63558c-0214-4e85-a413-d13728cd5e3f" containerName="revoke-keys"
Nov 01 23:23:31.667545 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.667444    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a63558c-0214-4e85-a413-d13728cd5e3f" containerName="revoke-keys"
Nov 01 23:23:31.667606 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.667557    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a63558c-0214-4e85-a413-d13728cd5e3f" containerName="revoke-keys"
Nov 01 23:23:31.670944 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.670923    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.726211 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.726187    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"]
Nov 01 23:23:31.768908 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.768884    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:23:31.769108 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.769091    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:23:31.813435 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.813411    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmf8d\" (UniqueName: \"kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.813535 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.813452    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.813581 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.813557    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.914550 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.914523    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmf8d\" (UniqueName: \"kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.914652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.914562    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.914652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.914636    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.915290 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.915268    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.916866 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.916826    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.969242 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.969224    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmf8d\" (UniqueName: \"kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d\") pod \"maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:31.983968 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:31.983946    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:32.353106 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:32.353073    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"]
Nov 01 23:23:32.361548 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:23:32.361515    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13a0d77_917b_467d_a849_4b624c0bc9e9.slice/crio-0f9beeedab54b345c3a7fafc7ac15ed35060210d0778204b22718401c380188e WatchSource:0}: Error finding container 0f9beeedab54b345c3a7fafc7ac15ed35060210d0778204b22718401c380188e: Status 404 returned error can't find the container with id 0f9beeedab54b345c3a7fafc7ac15ed35060210d0778204b22718401c380188e
Nov 01 23:23:32.971848 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:32.971814    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2" event={"ID":"a13a0d77-917b-467d-a849-4b624c0bc9e9","Type":"ContainerStarted","Data":"0dee44f0e04856b63eae5430c9f6f177c3da642d744122d59b5e6e5a63dfa627"}
Nov 01 23:23:32.972325 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:32.971871    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2" event={"ID":"a13a0d77-917b-467d-a849-4b624c0bc9e9","Type":"ContainerStarted","Data":"0f9beeedab54b345c3a7fafc7ac15ed35060210d0778204b22718401c380188e"}
Nov 01 23:23:33.079558 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:33.079509    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2" podStartSLOduration=2.079495585 podStartE2EDuration="2.079495585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:23:33.078706676 +0000 UTC m=+3594.396356677" watchObservedRunningTime="2026-11-01 23:23:33.079495585 +0000 UTC m=+3594.397145585"
Nov 01 23:23:33.978473 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:33.978435    4531 generic.go:356] "Generic (PLEG): container finished" podID="a13a0d77-917b-467d-a849-4b624c0bc9e9" containerID="0dee44f0e04856b63eae5430c9f6f177c3da642d744122d59b5e6e5a63dfa627" exitCode=0
Nov 01 23:23:33.978897 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:33.978526    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2" event={"ID":"a13a0d77-917b-467d-a849-4b624c0bc9e9","Type":"ContainerDied","Data":"0dee44f0e04856b63eae5430c9f6f177c3da642d744122d59b5e6e5a63dfa627"}
Nov 01 23:23:35.107330 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.107303    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:35.246654 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.246567    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d\" (UniqueName: \"kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d\") pod \"a13a0d77-917b-467d-a849-4b624c0bc9e9\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") "
Nov 01 23:23:35.246859 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.246706    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca\") pod \"a13a0d77-917b-467d-a849-4b624c0bc9e9\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") "
Nov 01 23:23:35.246859 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.246745    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca\") pod \"a13a0d77-917b-467d-a849-4b624c0bc9e9\" (UID: \"a13a0d77-917b-467d-a849-4b624c0bc9e9\") "
Nov 01 23:23:35.247172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.247142    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca" pod "a13a0d77-917b-467d-a849-4b624c0bc9e9" (UID: "a13a0d77-917b-467d-a849-4b624c0bc9e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:23:35.248857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.248833    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca" pod "a13a0d77-917b-467d-a849-4b624c0bc9e9" (UID: "a13a0d77-917b-467d-a849-4b624c0bc9e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:23:35.249275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.249258    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d" pod "a13a0d77-917b-467d-a849-4b624c0bc9e9" (UID: "a13a0d77-917b-467d-a849-4b624c0bc9e9"). InnerVolumeSpecName "kube-api-access-hmf8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:23:35.348364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.348322    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:35.348364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.348365    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13a0d77-917b-467d-a849-4b624c0bc9e9-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:35.348546 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.348384    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmf8d\" (UniqueName: \"kubernetes.io/projected/a13a0d77-917b-467d-a849-4b624c0bc9e9-kube-api-access-hmf8d\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:23:35.821440 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.821405    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:35.992358 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.992332    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"
Nov 01 23:23:35.992569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.992328    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2" event={"ID":"a13a0d77-917b-467d-a849-4b624c0bc9e9","Type":"ContainerDied","Data":"0f9beeedab54b345c3a7fafc7ac15ed35060210d0778204b22718401c380188e"}
Nov 01 23:23:35.992569 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:35.992468    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f9beeedab54b345c3a7fafc7ac15ed35060210d0778204b22718401c380188e"
Nov 01 23:23:36.210603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:36.210568    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:40.606652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:40.606575    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:42.526860 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:23:42.526823    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:23:43.587387 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:43.587339    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:43.768593 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:43.768560    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:44.789961 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:44.789924    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:48.069107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:48.069073    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:52.151005 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:52.150967    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:52.435926 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:52.435818    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:52.708279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:52.708245    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"]
Nov 01 23:23:52.751722 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:52.751693    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-3fc07c-5dbg2"]
Nov 01 23:23:52.814788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:52.814763    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13a0d77-917b-467d-a849-4b624c0bc9e9" path="/var/lib/kubelet/pods/a13a0d77-917b-467d-a849-4b624c0bc9e9/volumes"
Nov 01 23:23:53.176262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:53.176188    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:58.868914 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:58.868875    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:59.054483 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:59.054445    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:23:59.261245 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:23:59.261185    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:24:01.046609 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:01.046572    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:24:01.284560 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:01.284530    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:24:06.789792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:06.789761    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:24:12.526014 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:24:12.525971    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:24:19.466561 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.466370    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"]
Nov 01 23:24:19.467062 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.466679    4531 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88" podUID="85553d96-8d53-4f99-9b37-e5ada18e126e" containerName="manager" containerID="cri-o://927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6" gracePeriod=10
Nov 01 23:24:19.731915 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.731855    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 23:24:19.844972 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.844934    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume\") pod \"85553d96-8d53-4f99-9b37-e5ada18e126e\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") "
Nov 01 23:24:19.845117 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.845051    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb\" (UniqueName: \"kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb\") pod \"85553d96-8d53-4f99-9b37-e5ada18e126e\" (UID: \"85553d96-8d53-4f99-9b37-e5ada18e126e\") "
Nov 01 23:24:19.845345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.845321    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume" pod "85553d96-8d53-4f99-9b37-e5ada18e126e" (UID: "85553d96-8d53-4f99-9b37-e5ada18e126e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 23:24:19.847213 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.847190    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb" pod "85553d96-8d53-4f99-9b37-e5ada18e126e" (UID: "85553d96-8d53-4f99-9b37-e5ada18e126e"). InnerVolumeSpecName "kube-api-access-fczhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:24:19.946746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.946673    4531 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/85553d96-8d53-4f99-9b37-e5ada18e126e-extensions-socket-volume\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:19.946746 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:19.946716    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fczhb\" (UniqueName: \"kubernetes.io/projected/85553d96-8d53-4f99-9b37-e5ada18e126e-kube-api-access-fczhb\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:24:20.255825 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.255792    4531 generic.go:356] "Generic (PLEG): container finished" podID="85553d96-8d53-4f99-9b37-e5ada18e126e" containerID="927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6" exitCode=0
Nov 01 23:24:20.256003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.255837    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"
Nov 01 23:24:20.256003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.255864    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88" event={"ID":"85553d96-8d53-4f99-9b37-e5ada18e126e","Type":"ContainerDied","Data":"927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6"}
Nov 01 23:24:20.256003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.255905    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88" event={"ID":"85553d96-8d53-4f99-9b37-e5ada18e126e","Type":"ContainerDied","Data":"a2117886ea086d85ab1107b773a1daadf9efa7b64f634a998668bb726d9f5551"}
Nov 01 23:24:20.256003 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.255927    4531 scope.go:122] "RemoveContainer" containerID="927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6"
Nov 01 23:24:20.266779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.266761    4531 scope.go:122] "RemoveContainer" containerID="927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6"
Nov 01 23:24:20.267069 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:24:20.267051    4531 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6\": container with ID starting with 927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6 not found: ID does not exist" containerID="927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6"
Nov 01 23:24:20.267157 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.267077    4531 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6"} err="failed to get container status \"927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6\": rpc error: code = NotFound desc = could not find container \"927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6\": container with ID starting with 927221d290fb8f8ff45f9c96feed63e9dcf6737e767233de7a44b43e3f4698d6 not found: ID does not exist"
Nov 01 23:24:20.409526 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.409502    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"]
Nov 01 23:24:20.456108 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.456084    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-tcx88"]
Nov 01 23:24:20.815167 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:20.815140    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85553d96-8d53-4f99-9b37-e5ada18e126e" path="/var/lib/kubelet/pods/85553d96-8d53-4f99-9b37-e5ada18e126e/volumes"
Nov 01 23:24:42.525384 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:24:42.525346    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:24:48.527023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:24:48.526989    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:25:12.525218 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:25:12.525184    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:25:23.089814 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.089784    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"]
Nov 01 23:25:23.090222 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.090207    4531 cpu_manager.go:414] "containerMap: removing container" podUID="85553d96-8d53-4f99-9b37-e5ada18e126e" containerName="manager"
Nov 01 23:25:23.090269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.090223    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85553d96-8d53-4f99-9b37-e5ada18e126e" containerName="manager"
Nov 01 23:25:23.090269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.090233    4531 cpu_manager.go:414] "containerMap: removing container" podUID="a13a0d77-917b-467d-a849-4b624c0bc9e9" containerName="revoke-keys"
Nov 01 23:25:23.090269 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.090239    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13a0d77-917b-467d-a849-4b624c0bc9e9" containerName="revoke-keys"
Nov 01 23:25:23.090372 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.090303    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="85553d96-8d53-4f99-9b37-e5ada18e126e" containerName="manager"
Nov 01 23:25:23.090372 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.090311    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13a0d77-917b-467d-a849-4b624c0bc9e9" containerName="revoke-keys"
Nov 01 23:25:23.093490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.093471    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.144481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.144455    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"]
Nov 01 23:25:23.167230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.167205    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bfafb5-ea72-446c-8a1f-b8ea8172020d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7xs5r\" (UID: \"07bfafb5-ea72-446c-8a1f-b8ea8172020d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.167355 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.167235    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h54jm\" (UniqueName: \"kubernetes.io/projected/07bfafb5-ea72-446c-8a1f-b8ea8172020d-kube-api-access-h54jm\") pod \"kuadrant-operator-controller-manager-678d87766f-7xs5r\" (UID: \"07bfafb5-ea72-446c-8a1f-b8ea8172020d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.188006 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.187986    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-fbpnd\""
Nov 01 23:25:23.267859 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.267834    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bfafb5-ea72-446c-8a1f-b8ea8172020d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7xs5r\" (UID: \"07bfafb5-ea72-446c-8a1f-b8ea8172020d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.267970 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.267868    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h54jm\" (UniqueName: \"kubernetes.io/projected/07bfafb5-ea72-446c-8a1f-b8ea8172020d-kube-api-access-h54jm\") pod \"kuadrant-operator-controller-manager-678d87766f-7xs5r\" (UID: \"07bfafb5-ea72-446c-8a1f-b8ea8172020d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.268180 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.268161    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bfafb5-ea72-446c-8a1f-b8ea8172020d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-7xs5r\" (UID: \"07bfafb5-ea72-446c-8a1f-b8ea8172020d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.329511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.329487    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h54jm\" (UniqueName: \"kubernetes.io/projected/07bfafb5-ea72-446c-8a1f-b8ea8172020d-kube-api-access-h54jm\") pod \"kuadrant-operator-controller-manager-678d87766f-7xs5r\" (UID: \"07bfafb5-ea72-446c-8a1f-b8ea8172020d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.406408 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.406361    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:23.887261 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:23.887233    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"]
Nov 01 23:25:23.890910 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:25:23.890883    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bfafb5_ea72_446c_8a1f_b8ea8172020d.slice/crio-c9cbfe0924c11d2f08d596f9a761cc4f4e56c115c2fb8bbc6ff9c8501f994623 WatchSource:0}: Error finding container c9cbfe0924c11d2f08d596f9a761cc4f4e56c115c2fb8bbc6ff9c8501f994623: Status 404 returned error can't find the container with id c9cbfe0924c11d2f08d596f9a761cc4f4e56c115c2fb8bbc6ff9c8501f994623
Nov 01 23:25:24.621411 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:24.621373    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r" event={"ID":"07bfafb5-ea72-446c-8a1f-b8ea8172020d","Type":"ContainerStarted","Data":"cacfda5bd3194626dd54efdaf3afd3843dee875ffc1c8048a628ecf9f139e42f"}
Nov 01 23:25:24.621860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:24.621432    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r" event={"ID":"07bfafb5-ea72-446c-8a1f-b8ea8172020d","Type":"ContainerStarted","Data":"c9cbfe0924c11d2f08d596f9a761cc4f4e56c115c2fb8bbc6ff9c8501f994623"}
Nov 01 23:25:24.621860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:24.621602    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:24.827691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:24.827643    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r" podStartSLOduration=1.827629762 podStartE2EDuration="1.827629762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:25:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:25:24.781180113 +0000 UTC m=+3706.098830114" watchObservedRunningTime="2026-11-01 23:25:24.827629762 +0000 UTC m=+3706.145279764"
Nov 01 23:25:35.629461 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:35.629433    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-7xs5r"
Nov 01 23:25:42.525283 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:25:42.525244    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:25:48.755136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:25:48.755092    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:12.526510 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:26:12.526470    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:26:30.370005 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:30.369972    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:30.932220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:30.932186    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:31.160821 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:31.160713    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:35.157279 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:35.157248    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:35.427490 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:35.427415    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:38.430704 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:38.430671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:38.616580 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:38.616549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:41.457056 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:41.457006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:41.642021 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:41.641984    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:42.525104 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:26:42.525067    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:26:46.398147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:46.398112    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:49.105419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:49.105374    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:51.287393 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:51.287326    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:56.098736 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:56.098697    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:26:59.969047 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:26:59.969011    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:00.544329 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:00.544296    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:00.745564 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:00.745528    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:01.212904 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:01.212867    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:03.093486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:03.093442    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:05.181131 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:05.181091    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:05.993937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:05.993898    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:10.897226 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:10.897193    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:12.526081 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:27:12.526048    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:27:15.702901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:15.702856    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:20.557648 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:20.557565    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:25.300064 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:25.300028    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:30.086878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:30.086845    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:34.904414 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:34.904381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:39.687397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:39.687350    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:42.525068 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:27:42.525032    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:27:44.541997 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:44.541959    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:48.556054 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.556014    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"]
Nov 01 23:27:48.559792 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.559775    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.612808 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.612780    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"]
Nov 01 23:27:48.693127 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.693099    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:27:48.693248 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.693156    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:27:48.711200 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.711172    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvsst\" (UniqueName: \"kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.711311 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.711212    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.711311 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.711283    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.812130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.812064    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvsst\" (UniqueName: \"kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.812130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.812106    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.812299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.812149    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.812819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.812795    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.814582 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.814555    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.865107 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.865084    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvsst\" (UniqueName: \"kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst\") pod \"maas-api-revoke-sub-cfe661c5-wfh9f\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:48.871928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:48.871910    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:49.245146 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:49.245117    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"]
Nov 01 23:27:49.250257 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:27:49.250230    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84f84877_c830_4bea_a032_4eda2423aca9.slice/crio-4b75a6c9a02687fc459c4e6faaf27530f2e964ce1aa54b51de1a319e151b808f WatchSource:0}: Error finding container 4b75a6c9a02687fc459c4e6faaf27530f2e964ce1aa54b51de1a319e151b808f: Status 404 returned error can't find the container with id 4b75a6c9a02687fc459c4e6faaf27530f2e964ce1aa54b51de1a319e151b808f
Nov 01 23:27:49.391426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:49.391388    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:49.479873 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:49.479835    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f" event={"ID":"84f84877-c830-4bea-a032-4eda2423aca9","Type":"ContainerStarted","Data":"4b75a6c9a02687fc459c4e6faaf27530f2e964ce1aa54b51de1a319e151b808f"}
Nov 01 23:27:49.568051 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:49.568027    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:50.490221 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:50.490189    4531 generic.go:356] "Generic (PLEG): container finished" podID="84f84877-c830-4bea-a032-4eda2423aca9" containerID="53db727796bf99829d02818b6cd089301b2cf94c51fcd4cb9a9c9fd3934e9862" exitCode=0
Nov 01 23:27:50.490438 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:50.490235    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f" event={"ID":"84f84877-c830-4bea-a032-4eda2423aca9","Type":"ContainerDied","Data":"53db727796bf99829d02818b6cd089301b2cf94c51fcd4cb9a9c9fd3934e9862"}
Nov 01 23:27:51.716962 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.716925    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:51.743996 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.743967    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst\" (UniqueName: \"kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst\") pod \"84f84877-c830-4bea-a032-4eda2423aca9\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") "
Nov 01 23:27:51.744141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.744020    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca\") pod \"84f84877-c830-4bea-a032-4eda2423aca9\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") "
Nov 01 23:27:51.744141 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.744048    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca\") pod \"84f84877-c830-4bea-a032-4eda2423aca9\" (UID: \"84f84877-c830-4bea-a032-4eda2423aca9\") "
Nov 01 23:27:51.744424 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.744399    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca" pod "84f84877-c830-4bea-a032-4eda2423aca9" (UID: "84f84877-c830-4bea-a032-4eda2423aca9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:27:51.746223 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.746197    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca" pod "84f84877-c830-4bea-a032-4eda2423aca9" (UID: "84f84877-c830-4bea-a032-4eda2423aca9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:27:51.746322 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.746279    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst" pod "84f84877-c830-4bea-a032-4eda2423aca9" (UID: "84f84877-c830-4bea-a032-4eda2423aca9"). InnerVolumeSpecName "kube-api-access-qvsst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:27:51.846433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.846077    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvsst\" (UniqueName: \"kubernetes.io/projected/84f84877-c830-4bea-a032-4eda2423aca9-kube-api-access-qvsst\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:27:51.846433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.846111    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:27:51.846433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:51.846127    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84f84877-c830-4bea-a032-4eda2423aca9-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:27:52.516177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:52.516140    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f" event={"ID":"84f84877-c830-4bea-a032-4eda2423aca9","Type":"ContainerDied","Data":"4b75a6c9a02687fc459c4e6faaf27530f2e964ce1aa54b51de1a319e151b808f"}
Nov 01 23:27:52.516364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:52.516187    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"
Nov 01 23:27:52.516364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:52.516216    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b75a6c9a02687fc459c4e6faaf27530f2e964ce1aa54b51de1a319e151b808f"
Nov 01 23:27:54.203174 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:54.203137    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:58.992884 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:58.992852    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:27:59.699938 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:27:59.699907    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:00.139668 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.139591    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"]
Nov 01 23:28:00.140132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.140095    4531 cpu_manager.go:414] "containerMap: removing container" podUID="84f84877-c830-4bea-a032-4eda2423aca9" containerName="revoke-keys"
Nov 01 23:28:00.140132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.140115    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84f84877-c830-4bea-a032-4eda2423aca9" containerName="revoke-keys"
Nov 01 23:28:00.140268 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.140209    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="84f84877-c830-4bea-a032-4eda2423aca9" containerName="revoke-keys"
Nov 01 23:28:00.143642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.143625    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.199960 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.199935    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"]
Nov 01 23:28:00.213538 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.213508    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.213699 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.213561    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl2b4\" (UniqueName: \"kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.213819 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.213736    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.242081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.242030    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:28:00.242081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.242061    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:28:00.314834 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.314807    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.314965 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.314857    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl2b4\" (UniqueName: \"kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.315014 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.314959    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.315502 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.315473    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.317275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.317257    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.367670 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.367645    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl2b4\" (UniqueName: \"kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4\") pod \"maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.457259 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.457233    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:00.825460 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:00.825384    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"]
Nov 01 23:28:00.830437 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:28:00.830407    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51ca165_17f0_40bf_9b7b_aba15f057b55.slice/crio-b28cc5d8b5b20332960f30f09c21578b2f83eb11dd1bf7c8dc57f7c9a5fe49c9 WatchSource:0}: Error finding container b28cc5d8b5b20332960f30f09c21578b2f83eb11dd1bf7c8dc57f7c9a5fe49c9: Status 404 returned error can't find the container with id b28cc5d8b5b20332960f30f09c21578b2f83eb11dd1bf7c8dc57f7c9a5fe49c9
Nov 01 23:28:01.577743 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:01.577692    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m" event={"ID":"b51ca165-17f0-40bf-9b7b-aba15f057b55","Type":"ContainerStarted","Data":"c51358bef9277e2a81e40fde81ea34aed090a6c59682e2f3d9ea00821e4d3153"}
Nov 01 23:28:01.578220 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:01.577786    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m" event={"ID":"b51ca165-17f0-40bf-9b7b-aba15f057b55","Type":"ContainerStarted","Data":"b28cc5d8b5b20332960f30f09c21578b2f83eb11dd1bf7c8dc57f7c9a5fe49c9"}
Nov 01 23:28:01.684646 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:01.684567    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m" podStartSLOduration=1.684554383 podStartE2EDuration="1.684554383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:28:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:28:01.68365371 +0000 UTC m=+3863.001303711" watchObservedRunningTime="2026-11-01 23:28:01.684554383 +0000 UTC m=+3863.002204383"
Nov 01 23:28:02.585048 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:02.585010    4531 generic.go:356] "Generic (PLEG): container finished" podID="b51ca165-17f0-40bf-9b7b-aba15f057b55" containerID="c51358bef9277e2a81e40fde81ea34aed090a6c59682e2f3d9ea00821e4d3153" exitCode=0
Nov 01 23:28:02.585426 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:02.585091    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m" event={"ID":"b51ca165-17f0-40bf-9b7b-aba15f057b55","Type":"ContainerDied","Data":"c51358bef9277e2a81e40fde81ea34aed090a6c59682e2f3d9ea00821e4d3153"}
Nov 01 23:28:03.724599 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.724576    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:03.748515 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.748493    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4\" (UniqueName: \"kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4\") pod \"b51ca165-17f0-40bf-9b7b-aba15f057b55\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") "
Nov 01 23:28:03.748603 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.748581    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca\") pod \"b51ca165-17f0-40bf-9b7b-aba15f057b55\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") "
Nov 01 23:28:03.748651 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.748620    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca\") pod \"b51ca165-17f0-40bf-9b7b-aba15f057b55\" (UID: \"b51ca165-17f0-40bf-9b7b-aba15f057b55\") "
Nov 01 23:28:03.749037 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.749007    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca" pod "b51ca165-17f0-40bf-9b7b-aba15f057b55" (UID: "b51ca165-17f0-40bf-9b7b-aba15f057b55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:28:03.750524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.750502    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4" pod "b51ca165-17f0-40bf-9b7b-aba15f057b55" (UID: "b51ca165-17f0-40bf-9b7b-aba15f057b55"). InnerVolumeSpecName "kube-api-access-vl2b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:28:03.750610 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.750500    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca" pod "b51ca165-17f0-40bf-9b7b-aba15f057b55" (UID: "b51ca165-17f0-40bf-9b7b-aba15f057b55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:28:03.849433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.849350    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl2b4\" (UniqueName: \"kubernetes.io/projected/b51ca165-17f0-40bf-9b7b-aba15f057b55-kube-api-access-vl2b4\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:03.849433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.849382    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:03.849433 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:03.849396    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b51ca165-17f0-40bf-9b7b-aba15f057b55-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:28:04.100322 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:04.100110    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:04.598132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:04.598100    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"
Nov 01 23:28:04.598132 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:04.598125    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m" event={"ID":"b51ca165-17f0-40bf-9b7b-aba15f057b55","Type":"ContainerDied","Data":"b28cc5d8b5b20332960f30f09c21578b2f83eb11dd1bf7c8dc57f7c9a5fe49c9"}
Nov 01 23:28:04.598364 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:04.598196    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b28cc5d8b5b20332960f30f09c21578b2f83eb11dd1bf7c8dc57f7c9a5fe49c9"
Nov 01 23:28:04.733161 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:04.733134    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:08.893139 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:08.893101    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:12.056353 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:12.056314    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:12.255709 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:12.255676    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:12.525470 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:28:12.525437    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:28:13.697750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:13.697680    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:17.891166 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:17.891135    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:19.463578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:19.463541    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:19.741308 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:19.741141    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:20.021264 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:20.021173    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"]
Nov 01 23:28:20.066742 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:20.066690    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-0cff29-xr42m"]
Nov 01 23:28:20.475190 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:20.475156    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:20.815477 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:20.815392    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b51ca165-17f0-40bf-9b7b-aba15f057b55" path="/var/lib/kubelet/pods/b51ca165-17f0-40bf-9b7b-aba15f057b55/volumes"
Nov 01 23:28:24.047423 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:24.047390    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"]
Nov 01 23:28:24.093779 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:24.093750    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93a8cb84-vlxjt"]
Nov 01 23:28:24.815221 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:24.815188    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a63558c-0214-4e85-a413-d13728cd5e3f" path="/var/lib/kubelet/pods/2a63558c-0214-4e85-a413-d13728cd5e3f/volumes"
Nov 01 23:28:26.064105 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:26.064070    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:26.278828 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:26.278795    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:26.495053 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:26.495018    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:32.895440 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:32.895398    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:40.845529 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:40.845493    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:41.939989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:41.939954    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:42.524611 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:28:42.524576    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:28:50.152022 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:50.151939    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:28:56.316927 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:28:56.316887    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:29:02.565299 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:02.565264    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:29:02.760422 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:02.760391    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:29:12.524468 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:29:12.524431    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:29:16.210855 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:16.210820    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:29:42.525128 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:29:42.525091    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:29:42.695838 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:42.695807    4531 scope.go:122] "RemoveContainer" containerID="0dee44f0e04856b63eae5430c9f6f177c3da642d744122d59b5e6e5a63dfa627"
Nov 01 23:29:42.704931 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:42.704911    4531 scope.go:122] "RemoveContainer" containerID="3fecd1c68d9af883dc35da053b530f78cf53e401c35ce9ccbdf20e86f94ca1c9"
Nov 01 23:29:50.086963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:50.086918    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:29:53.192751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:53.185850    4531 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7f694df8bc-4txmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 23:29:53.192751 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:29:53.185916    4531 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7f694df8bc-4txmj" podUID="31480339-9503-4930-bfb6-3d8c4b2680ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 23:30:02.567267 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:02.567224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:30:07.172853 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:07.172810    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:30:11.178136 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.178098    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"]
Nov 01 23:30:11.178540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.178527    4531 cpu_manager.go:414] "containerMap: removing container" podUID="b51ca165-17f0-40bf-9b7b-aba15f057b55" containerName="revoke-keys"
Nov 01 23:30:11.178540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.178538    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b51ca165-17f0-40bf-9b7b-aba15f057b55" containerName="revoke-keys"
Nov 01 23:30:11.178647 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.178636    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="b51ca165-17f0-40bf-9b7b-aba15f057b55" containerName="revoke-keys"
Nov 01 23:30:11.181723 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.181704    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.278345 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.278317    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bf72t\""
Nov 01 23:30:11.278469 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.278352    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:30:11.281231 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.281214    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"]
Nov 01 23:30:11.353578 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.353553    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.353691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.353594    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.353691 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.353619    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7nrk\" (UniqueName: \"kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.454172 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.454144    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7nrk\" (UniqueName: \"kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.454317 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.454276    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.454385 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.454333    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.455014 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.454990    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.457050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.457028    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.509963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.509935    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7nrk\" (UniqueName: \"kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk\") pod \"maas-api-revoke-sub-3e31a506-5zxxm\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:11.794718 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:11.794641    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:12.167013 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:12.166919    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"]
Nov 01 23:30:12.172512 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:30:12.172481    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe65171_abe0_401b_b58a_c01a1fbf03a8.slice/crio-ba8e1dcd3462d4554af7c1d866c6e8ce27be224c4e9d9a1c954b09d3fffa6963 WatchSource:0}: Error finding container ba8e1dcd3462d4554af7c1d866c6e8ce27be224c4e9d9a1c954b09d3fffa6963: Status 404 returned error can't find the container with id ba8e1dcd3462d4554af7c1d866c6e8ce27be224c4e9d9a1c954b09d3fffa6963
Nov 01 23:30:12.469173 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:12.469137    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm" event={"ID":"1fe65171-abe0-401b-b58a-c01a1fbf03a8","Type":"ContainerStarted","Data":"ba8e1dcd3462d4554af7c1d866c6e8ce27be224c4e9d9a1c954b09d3fffa6963"}
Nov 01 23:30:12.539993 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:30:12.539948    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:30:13.513750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:13.503409    4531 generic.go:356] "Generic (PLEG): container finished" podID="1fe65171-abe0-401b-b58a-c01a1fbf03a8" containerID="b231eaf1531f184a16a7e69b4730830b83a0fa8970bb84c6d1b224bcae3ee583" exitCode=0
Nov 01 23:30:13.513750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:13.503581    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm" event={"ID":"1fe65171-abe0-401b-b58a-c01a1fbf03a8","Type":"ContainerDied","Data":"b231eaf1531f184a16a7e69b4730830b83a0fa8970bb84c6d1b224bcae3ee583"}
Nov 01 23:30:14.935063 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.935039    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:14.952897 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.952871    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca\") pod \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") "
Nov 01 23:30:14.953034 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.952963    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk\" (UniqueName: \"kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk\") pod \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") "
Nov 01 23:30:14.953034 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.952983    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca\") pod \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\" (UID: \"1fe65171-abe0-401b-b58a-c01a1fbf03a8\") "
Nov 01 23:30:14.953271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.953246    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca" pod "1fe65171-abe0-401b-b58a-c01a1fbf03a8" (UID: "1fe65171-abe0-401b-b58a-c01a1fbf03a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:30:14.955014 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.954986    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca" pod "1fe65171-abe0-401b-b58a-c01a1fbf03a8" (UID: "1fe65171-abe0-401b-b58a-c01a1fbf03a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:30:14.955118 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:14.955060    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk" pod "1fe65171-abe0-401b-b58a-c01a1fbf03a8" (UID: "1fe65171-abe0-401b-b58a-c01a1fbf03a8"). InnerVolumeSpecName "kube-api-access-p7nrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:30:15.053980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:15.053946    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:15.053980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:15.053974    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7nrk\" (UniqueName: \"kubernetes.io/projected/1fe65171-abe0-401b-b58a-c01a1fbf03a8-kube-api-access-p7nrk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:15.053980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:15.053983    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fe65171-abe0-401b-b58a-c01a1fbf03a8-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:30:15.543120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:15.543092    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"
Nov 01 23:30:15.543286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:15.543122    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm" event={"ID":"1fe65171-abe0-401b-b58a-c01a1fbf03a8","Type":"ContainerDied","Data":"ba8e1dcd3462d4554af7c1d866c6e8ce27be224c4e9d9a1c954b09d3fffa6963"}
Nov 01 23:30:15.543286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:15.543173    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba8e1dcd3462d4554af7c1d866c6e8ce27be224c4e9d9a1c954b09d3fffa6963"
Nov 01 23:30:42.525090 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:30:42.525045    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:30:50.426181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:30:50.426142    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:00.243457 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:00.243422    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:00.854526 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:00.854485    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:01.106199 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:01.106115    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:03.938511 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:03.938475    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:04.173902 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:04.173869    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:07.268053 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:07.268014    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:07.475567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:07.475532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:09.507702 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:09.507653    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:09.711275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:09.711219    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:09.906980 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:09.906886    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:12.532898 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:31:12.532859    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:31:14.216397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:14.216365    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:14.722650 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:14.722602    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:19.628251 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:19.627440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:24.428427 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:24.428389    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:26.271162 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:26.271127    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:29.590261 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:29.590224    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:34.623769 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:34.623715    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:39.553436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:39.553400    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:40.210133 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.210100    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"]
Nov 01 23:31:40.210654 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.210635    4531 cpu_manager.go:414] "containerMap: removing container" podUID="1fe65171-abe0-401b-b58a-c01a1fbf03a8" containerName="revoke-keys"
Nov 01 23:31:40.210654 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.210654    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe65171-abe0-401b-b58a-c01a1fbf03a8" containerName="revoke-keys"
Nov 01 23:31:40.210887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.210815    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe65171-abe0-401b-b58a-c01a1fbf03a8" containerName="revoke-keys"
Nov 01 23:31:40.214028 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.214013    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.266016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.265996    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ljx4x\""
Nov 01 23:31:40.266016 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.265995    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:31:40.269788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.269763    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"]
Nov 01 23:31:40.366085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.366053    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjcb\" (UniqueName: \"kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.366241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.366106    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.366241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.366219    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.467489 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.467422    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.467489 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.467455    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjcb\" (UniqueName: \"kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.467676 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.467493    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.468110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.468086    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.469709 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.469682    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.522022 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.521999    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjcb\" (UniqueName: \"kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb\") pod \"maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.526802 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.526786    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:40.894672 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:40.894602    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"]
Nov 01 23:31:40.899534 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:31:40.899498    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5d244f_c8ff_40f1_aea1_bed3c135cc92.slice/crio-3da32c5c8542fda01c46f59a72ee67caf880ae9a3e4c886691c77b5dd0ff466e WatchSource:0}: Error finding container 3da32c5c8542fda01c46f59a72ee67caf880ae9a3e4c886691c77b5dd0ff466e: Status 404 returned error can't find the container with id 3da32c5c8542fda01c46f59a72ee67caf880ae9a3e4c886691c77b5dd0ff466e
Nov 01 23:31:41.068974 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:41.068945    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm" event={"ID":"0b5d244f-c8ff-40f1-aea1-bed3c135cc92","Type":"ContainerStarted","Data":"3da32c5c8542fda01c46f59a72ee67caf880ae9a3e4c886691c77b5dd0ff466e"}
Nov 01 23:31:42.075991 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:42.075910    4531 generic.go:356] "Generic (PLEG): container finished" podID="0b5d244f-c8ff-40f1-aea1-bed3c135cc92" containerID="f13ece793b02cd52a2b59fe041d4c229cc2e45684e16145501380461e5c99570" exitCode=0
Nov 01 23:31:42.076368 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:42.075989    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm" event={"ID":"0b5d244f-c8ff-40f1-aea1-bed3c135cc92","Type":"ContainerDied","Data":"f13ece793b02cd52a2b59fe041d4c229cc2e45684e16145501380461e5c99570"}
Nov 01 23:31:42.525054 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:31:42.525022    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:31:43.214261 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.214236    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:43.291138 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.291109    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb\" (UniqueName: \"kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb\") pod \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") "
Nov 01 23:31:43.291435 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.291174    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca\") pod \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") "
Nov 01 23:31:43.291435 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.291241    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca\") pod \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\" (UID: \"0b5d244f-c8ff-40f1-aea1-bed3c135cc92\") "
Nov 01 23:31:43.291551 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.291478    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca" pod "0b5d244f-c8ff-40f1-aea1-bed3c135cc92" (UID: "0b5d244f-c8ff-40f1-aea1-bed3c135cc92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:31:43.291601 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.291572    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:43.293106 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.293086    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca" pod "0b5d244f-c8ff-40f1-aea1-bed3c135cc92" (UID: "0b5d244f-c8ff-40f1-aea1-bed3c135cc92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:31:43.293162 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.293107    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb" pod "0b5d244f-c8ff-40f1-aea1-bed3c135cc92" (UID: "0b5d244f-c8ff-40f1-aea1-bed3c135cc92"). InnerVolumeSpecName "kube-api-access-mhjcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:31:43.392227 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.392170    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjcb\" (UniqueName: \"kubernetes.io/projected/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-kube-api-access-mhjcb\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:43.392227 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:43.392194    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b5d244f-c8ff-40f1-aea1-bed3c135cc92-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:31:44.087636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:44.087610    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"
Nov 01 23:31:44.087636 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:44.087618    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm" event={"ID":"0b5d244f-c8ff-40f1-aea1-bed3c135cc92","Type":"ContainerDied","Data":"3da32c5c8542fda01c46f59a72ee67caf880ae9a3e4c886691c77b5dd0ff466e"}
Nov 01 23:31:44.087867 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:44.087665    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3da32c5c8542fda01c46f59a72ee67caf880ae9a3e4c886691c77b5dd0ff466e"
Nov 01 23:31:44.450388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:44.450356    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:49.215263 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:49.215223    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:50.747039 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:50.747006    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:52.134022 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:52.133982    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:52.341119 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:52.341085    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:54.143877 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:54.143841    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:58.373177 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:58.373133    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:31:59.935103 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:31:59.935067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:00.201481 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:00.201446    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:00.370930 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:00.370890    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"]
Nov 01 23:32:00.417436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:00.417406    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a38c0c87-7z2tm"]
Nov 01 23:32:00.816275 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:00.816245    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b5d244f-c8ff-40f1-aea1-bed3c135cc92" path="/var/lib/kubelet/pods/0b5d244f-c8ff-40f1-aea1-bed3c135cc92/volumes"
Nov 01 23:32:01.233366 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:01.233332    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:02.443039 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:02.443004    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:03.528781 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:03.528723    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:03.754063 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:03.754025    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:07.825286 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:07.825256    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:08.030695 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:08.030659    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:11.122156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:11.122123    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:11.323110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:11.323067    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:12.525167 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:32:12.525137    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:32:15.242156 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:15.242120    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:15.705592 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:15.705549    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:20.303359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:20.303317    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:25.200707 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:25.200639    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:30.051653 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:30.051616    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:32.632359 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:32.632321    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:35.198763 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:35.198714    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:40.119088 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:40.119044    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:40.642936 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:40.642892    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:42.525002 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:32:42.524968    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:32:45.065824 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:45.065791    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:49.803070 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:49.803030    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:51.076081 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:51.076051    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:52.048013 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:52.047980    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"]
Nov 01 23:32:52.093887 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:52.093858    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cfe661c5-wfh9f"]
Nov 01 23:32:52.815866 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:52.815831    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84f84877-c830-4bea-a032-4eda2423aca9" path="/var/lib/kubelet/pods/84f84877-c830-4bea-a032-4eda2423aca9/volumes"
Nov 01 23:32:54.708120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:54.708084    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:32:59.503706 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:32:59.503671    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:04.510575 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:04.510543    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:09.318463 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:09.318420    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:12.525061 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:33:12.525024    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:33:14.207094 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:14.207057    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:19.028946 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:19.028911    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:23.938026 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:23.937990    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:28.713148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:28.713107    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:33.621143 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:33.621092    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:38.382006 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:38.381970    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:42.524972 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:33:42.524937    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:33:43.179632 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:43.179598    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:47.932383 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:47.932349    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:51.472097 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:51.472063    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:52.782616 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:52.782582    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:55.719655 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.719624    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"]
Nov 01 23:33:55.720130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.720113    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0b5d244f-c8ff-40f1-aea1-bed3c135cc92" containerName="revoke-keys"
Nov 01 23:33:55.720178 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.720132    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b5d244f-c8ff-40f1-aea1-bed3c135cc92" containerName="revoke-keys"
Nov 01 23:33:55.720362 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.720347    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b5d244f-c8ff-40f1-aea1-bed3c135cc92" containerName="revoke-keys"
Nov 01 23:33:55.723673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.723657    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.776888 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.776863    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"]
Nov 01 23:33:55.819341 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.819320    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:33:55.819452 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.819344    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ljx4x\""
Nov 01 23:33:55.820568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.820551    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.820620 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.820577    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.820661 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.820622    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvvzl\" (UniqueName: \"kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.922112 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.922092    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvvzl\" (UniqueName: \"kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.922294 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.922274    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.922348 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.922303    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.922923 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.922900    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.924632 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.924608    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:55.975334 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:55.975280    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvvzl\" (UniqueName: \"kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl\") pod \"maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:56.036567 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:56.036548    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:56.403594 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:56.403495    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"]
Nov 01 23:33:56.409018 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:33:56.408993    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3556aa38_8d07_4714_9ec2_082fc293acca.slice/crio-858ca072f34d5ec19a6e9ced0dbe4d6cd5fd8f060708044d8878b251c99aa264 WatchSource:0}: Error finding container 858ca072f34d5ec19a6e9ced0dbe4d6cd5fd8f060708044d8878b251c99aa264: Status 404 returned error can't find the container with id 858ca072f34d5ec19a6e9ced0dbe4d6cd5fd8f060708044d8878b251c99aa264
Nov 01 23:33:56.862428 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:56.862399    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs" event={"ID":"3556aa38-8d07-4714-9ec2-082fc293acca","Type":"ContainerStarted","Data":"95bbddf213afe0c5fe1d6ef7c9a7a75f336550b945532f7476333eb48802575a"}
Nov 01 23:33:56.862809 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:56.862451    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs" event={"ID":"3556aa38-8d07-4714-9ec2-082fc293acca","Type":"ContainerStarted","Data":"858ca072f34d5ec19a6e9ced0dbe4d6cd5fd8f060708044d8878b251c99aa264"}
Nov 01 23:33:56.969771 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:56.969716    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs" podStartSLOduration=1.969702296 podStartE2EDuration="1.969702296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:33:56.967324163 +0000 UTC m=+4218.284974165" watchObservedRunningTime="2026-11-01 23:33:56.969702296 +0000 UTC m=+4218.287352300"
Nov 01 23:33:57.729839 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:57.729699    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:33:57.869902 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:57.869869    4531 generic.go:356] "Generic (PLEG): container finished" podID="3556aa38-8d07-4714-9ec2-082fc293acca" containerID="95bbddf213afe0c5fe1d6ef7c9a7a75f336550b945532f7476333eb48802575a" exitCode=0
Nov 01 23:33:57.870295 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:57.869908    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs" event={"ID":"3556aa38-8d07-4714-9ec2-082fc293acca","Type":"ContainerDied","Data":"95bbddf213afe0c5fe1d6ef7c9a7a75f336550b945532f7476333eb48802575a"}
Nov 01 23:33:59.011615 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.011584    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:59.150989 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.150913    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca\") pod \"3556aa38-8d07-4714-9ec2-082fc293acca\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") "
Nov 01 23:33:59.151119 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.151025    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca\") pod \"3556aa38-8d07-4714-9ec2-082fc293acca\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") "
Nov 01 23:33:59.151119 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.151057    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl\" (UniqueName: \"kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl\") pod \"3556aa38-8d07-4714-9ec2-082fc293acca\" (UID: \"3556aa38-8d07-4714-9ec2-082fc293acca\") "
Nov 01 23:33:59.151233 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.151212    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca" pod "3556aa38-8d07-4714-9ec2-082fc293acca" (UID: "3556aa38-8d07-4714-9ec2-082fc293acca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:33:59.151477 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.151456    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:33:59.153056 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.153027    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl" pod "3556aa38-8d07-4714-9ec2-082fc293acca" (UID: "3556aa38-8d07-4714-9ec2-082fc293acca"). InnerVolumeSpecName "kube-api-access-jvvzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:33:59.153143 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.153080    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca" pod "3556aa38-8d07-4714-9ec2-082fc293acca" (UID: "3556aa38-8d07-4714-9ec2-082fc293acca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:33:59.252431 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.252406    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3556aa38-8d07-4714-9ec2-082fc293acca-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:33:59.252431 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.252430    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvvzl\" (UniqueName: \"kubernetes.io/projected/3556aa38-8d07-4714-9ec2-082fc293acca-kube-api-access-jvvzl\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:33:59.882805 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.882778    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"
Nov 01 23:33:59.882985 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.882780    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs" event={"ID":"3556aa38-8d07-4714-9ec2-082fc293acca","Type":"ContainerDied","Data":"858ca072f34d5ec19a6e9ced0dbe4d6cd5fd8f060708044d8878b251c99aa264"}
Nov 01 23:33:59.882985 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:33:59.882908    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="858ca072f34d5ec19a6e9ced0dbe4d6cd5fd8f060708044d8878b251c99aa264"
Nov 01 23:34:02.717485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:02.717444    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:07.501079 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:07.501036    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:07.772649 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:07.772562    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:07.949365 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:07.949326    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:12.525089 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:34:12.525054    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:34:16.785145 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:16.785106    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:18.263432 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:18.263381    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:18.540491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:18.540411    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:18.815630 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:18.815561    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"]
Nov 01 23:34:18.855858 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:18.855829    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-2474f7-kscgs"]
Nov 01 23:34:19.233523 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:19.233491    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:20.816460 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:20.816429    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3556aa38-8d07-4714-9ec2-082fc293acca" path="/var/lib/kubelet/pods/3556aa38-8d07-4714-9ec2-082fc293acca/volumes"
Nov 01 23:34:25.042893 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:25.042860    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:25.222120 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:25.222091    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:25.507065 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:25.507029    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:28.223937 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:28.223903    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:42.528829 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:34:42.528785    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:34:42.774324 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:42.774298    4531 scope.go:122] "RemoveContainer" containerID="c51358bef9277e2a81e40fde81ea34aed090a6c59682e2f3d9ea00821e4d3153"
Nov 01 23:34:42.783556 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:42.783532    4531 scope.go:122] "RemoveContainer" containerID="53db727796bf99829d02818b6cd089301b2cf94c51fcd4cb9a9c9fd3934e9862"
Nov 01 23:34:51.732605 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:51.732532    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:34:52.274977 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.274945    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"]
Nov 01 23:34:52.275388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.275373    4531 cpu_manager.go:414] "containerMap: removing container" podUID="3556aa38-8d07-4714-9ec2-082fc293acca" containerName="revoke-keys"
Nov 01 23:34:52.275388 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.275388    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3556aa38-8d07-4714-9ec2-082fc293acca" containerName="revoke-keys"
Nov 01 23:34:52.275522 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.275489    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="3556aa38-8d07-4714-9ec2-082fc293acca" containerName="revoke-keys"
Nov 01 23:34:52.278818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.278798    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.334188 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.334143    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"]
Nov 01 23:34:52.363007 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.362976    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.363148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.363046    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.363225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.363171    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkvsp\" (UniqueName: \"kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.387570 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.387538    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:34:52.387716 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.387632    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ljx4x\""
Nov 01 23:34:52.463585 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.463555    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.463748 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.463651    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkvsp\" (UniqueName: \"kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.463748 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.463687    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.464225 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.464204    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.466148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.466123    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.516642 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.516614    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkvsp\" (UniqueName: \"kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp\") pod \"maas-api-revoke-sub-f03629db-s65jg\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.591487 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.591438    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:52.958592 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:52.958562    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"]
Nov 01 23:34:52.963694 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:34:52.963664    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74f347d_671f_4320_9488_b5a92363d39d.slice/crio-f1e76b8be617d7e114df9d3521303535744b60447c253d1f17edd1065122c150 WatchSource:0}: Error finding container f1e76b8be617d7e114df9d3521303535744b60447c253d1f17edd1065122c150: Status 404 returned error can't find the container with id f1e76b8be617d7e114df9d3521303535744b60447c253d1f17edd1065122c150
Nov 01 23:34:53.260939 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:53.260867    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg" event={"ID":"f74f347d-671f-4320-9488-b5a92363d39d","Type":"ContainerStarted","Data":"f1e76b8be617d7e114df9d3521303535744b60447c253d1f17edd1065122c150"}
Nov 01 23:34:54.269072 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:54.269040    4531 generic.go:356] "Generic (PLEG): container finished" podID="f74f347d-671f-4320-9488-b5a92363d39d" containerID="41e725d2db6a4fe994a843860ab9299b5781492ce10d2c5252a5f01a5023ff01" exitCode=0
Nov 01 23:34:54.269485 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:54.269129    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg" event={"ID":"f74f347d-671f-4320-9488-b5a92363d39d","Type":"ContainerDied","Data":"41e725d2db6a4fe994a843860ab9299b5781492ce10d2c5252a5f01a5023ff01"}
Nov 01 23:34:55.411860 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.411837    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:55.590541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.590470    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca\") pod \"f74f347d-671f-4320-9488-b5a92363d39d\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") "
Nov 01 23:34:55.590541 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.590517    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp\" (UniqueName: \"kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp\") pod \"f74f347d-671f-4320-9488-b5a92363d39d\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") "
Nov 01 23:34:55.590818 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.590553    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca\") pod \"f74f347d-671f-4320-9488-b5a92363d39d\" (UID: \"f74f347d-671f-4320-9488-b5a92363d39d\") "
Nov 01 23:34:55.590901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.590876    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca" pod "f74f347d-671f-4320-9488-b5a92363d39d" (UID: "f74f347d-671f-4320-9488-b5a92363d39d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:34:55.592699 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.592676    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp" pod "f74f347d-671f-4320-9488-b5a92363d39d" (UID: "f74f347d-671f-4320-9488-b5a92363d39d"). InnerVolumeSpecName "kube-api-access-mkvsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:34:55.592699 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.592687    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca" pod "f74f347d-671f-4320-9488-b5a92363d39d" (UID: "f74f347d-671f-4320-9488-b5a92363d39d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:34:55.691986 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.691960    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:55.691986 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.691983    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkvsp\" (UniqueName: \"kubernetes.io/projected/f74f347d-671f-4320-9488-b5a92363d39d-kube-api-access-mkvsp\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:55.692154 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:55.691995    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f74f347d-671f-4320-9488-b5a92363d39d-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:34:56.281689 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:56.281664    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg"
Nov 01 23:34:56.281898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:56.281660    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-s65jg" event={"ID":"f74f347d-671f-4320-9488-b5a92363d39d","Type":"ContainerDied","Data":"f1e76b8be617d7e114df9d3521303535744b60447c253d1f17edd1065122c150"}
Nov 01 23:34:56.281898 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:34:56.281796    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1e76b8be617d7e114df9d3521303535744b60447c253d1f17edd1065122c150"
Nov 01 23:35:12.524682 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:35:12.524644    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:35:12.835025 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.834953    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"]
Nov 01 23:35:12.835619 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.835578    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f74f347d-671f-4320-9488-b5a92363d39d" containerName="revoke-keys"
Nov 01 23:35:12.835619 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.835605    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74f347d-671f-4320-9488-b5a92363d39d" containerName="revoke-keys"
Nov 01 23:35:12.835871 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.835759    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74f347d-671f-4320-9488-b5a92363d39d" containerName="revoke-keys"
Nov 01 23:35:12.839230 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.839202    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.863356 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.863326    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.863510 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.863374    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xbzk\" (UniqueName: \"kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.863581 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.863535    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.935331 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.935305    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:35:12.935482 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.935380    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ljx4x\""
Nov 01 23:35:12.938478 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.938455    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"]
Nov 01 23:35:12.964067 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.964043    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.964193 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.964111    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.964193 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.964137    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xbzk\" (UniqueName: \"kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.964696 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.964674    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:12.966419 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:12.966401    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:13.018060 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:13.018040    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xbzk\" (UniqueName: \"kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk\") pod \"maas-api-revoke-sub-0165d7d2-rsmxq\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:13.153159 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:13.153109    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:13.523607 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:13.523564    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"]
Nov 01 23:35:13.528108 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:35:13.528080    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf316c8b5_c1ff_4119_ae79_cf194fa662b0.slice/crio-85d114b7a41b879eb4c8bbbd15c97ec0018803ec8304428d62bf55fa401eee2b WatchSource:0}: Error finding container 85d114b7a41b879eb4c8bbbd15c97ec0018803ec8304428d62bf55fa401eee2b: Status 404 returned error can't find the container with id 85d114b7a41b879eb4c8bbbd15c97ec0018803ec8304428d62bf55fa401eee2b
Nov 01 23:35:14.405721 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:14.405687    4531 generic.go:356] "Generic (PLEG): container finished" podID="f316c8b5-c1ff-4119-ae79-cf194fa662b0" containerID="61d7421d9cf728ea6897e718059055209f58027d789bd9f9ea83e85d8fc48be9" exitCode=0
Nov 01 23:35:14.405857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:14.405760    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq" event={"ID":"f316c8b5-c1ff-4119-ae79-cf194fa662b0","Type":"ContainerDied","Data":"61d7421d9cf728ea6897e718059055209f58027d789bd9f9ea83e85d8fc48be9"}
Nov 01 23:35:14.405857 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:14.405823    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq" event={"ID":"f316c8b5-c1ff-4119-ae79-cf194fa662b0","Type":"ContainerStarted","Data":"85d114b7a41b879eb4c8bbbd15c97ec0018803ec8304428d62bf55fa401eee2b"}
Nov 01 23:35:15.551765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.551743    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:15.585697 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.585672    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca\") pod \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") "
Nov 01 23:35:15.585820 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.585722    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca\") pod \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") "
Nov 01 23:35:15.585863 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.585840    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk\" (UniqueName: \"kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk\") pod \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\" (UID: \"f316c8b5-c1ff-4119-ae79-cf194fa662b0\") "
Nov 01 23:35:15.586021 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.586001    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca" pod "f316c8b5-c1ff-4119-ae79-cf194fa662b0" (UID: "f316c8b5-c1ff-4119-ae79-cf194fa662b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:35:15.586148 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.586135    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:35:15.587621 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.587600    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca" pod "f316c8b5-c1ff-4119-ae79-cf194fa662b0" (UID: "f316c8b5-c1ff-4119-ae79-cf194fa662b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:35:15.587753 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.587712    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk" pod "f316c8b5-c1ff-4119-ae79-cf194fa662b0" (UID: "f316c8b5-c1ff-4119-ae79-cf194fa662b0"). InnerVolumeSpecName "kube-api-access-7xbzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:35:15.686918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.686855    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f316c8b5-c1ff-4119-ae79-cf194fa662b0-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:35:15.686918 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:15.686877    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xbzk\" (UniqueName: \"kubernetes.io/projected/f316c8b5-c1ff-4119-ae79-cf194fa662b0-kube-api-access-7xbzk\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:35:16.046436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:16.046412    4531 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"]
Nov 01 23:35:16.051750 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:16.049689    4531 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-5zxxm"]
Nov 01 23:35:16.419235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:16.419160    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq"
Nov 01 23:35:16.419235 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:16.419199    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-rsmxq" event={"ID":"f316c8b5-c1ff-4119-ae79-cf194fa662b0","Type":"ContainerDied","Data":"85d114b7a41b879eb4c8bbbd15c97ec0018803ec8304428d62bf55fa401eee2b"}
Nov 01 23:35:16.419519 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:16.419275    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d114b7a41b879eb4c8bbbd15c97ec0018803ec8304428d62bf55fa401eee2b"
Nov 01 23:35:16.815019 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:16.814993    4531 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe65171-abe0-401b-b58a-c01a1fbf03a8" path="/var/lib/kubelet/pods/1fe65171-abe0-401b-b58a-c01a1fbf03a8/volumes"
Nov 01 23:35:19.049829 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:19.049800    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:35:24.458257 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.458222    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"]
Nov 01 23:35:24.458910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.458877    4531 cpu_manager.go:414] "containerMap: removing container" podUID="f316c8b5-c1ff-4119-ae79-cf194fa662b0" containerName="revoke-keys"
Nov 01 23:35:24.458910 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.458898    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f316c8b5-c1ff-4119-ae79-cf194fa662b0" containerName="revoke-keys"
Nov 01 23:35:24.459085 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.459043    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="f316c8b5-c1ff-4119-ae79-cf194fa662b0" containerName="revoke-keys"
Nov 01 23:35:24.462665 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.462646    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.514027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.513998    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"]
Nov 01 23:35:24.556926 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.556902    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 23:35:24.557066 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.556944    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-ljx4x\""
Nov 01 23:35:24.559386 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.559360    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6tst\" (UniqueName: \"kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.559484 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.559397    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.559537 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.559481    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.660122 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.660097    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6tst\" (UniqueName: \"kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.660241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.660126    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.660241 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.660154    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.660883 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.660866    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.662458 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.662428    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.713765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.713694    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6tst\" (UniqueName: \"kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst\") pod \"maas-api-revoke-sub-6b19338f-rtp8b\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:24.775889 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:24.775864    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:25.148371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:25.148294    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"]
Nov 01 23:35:25.153262 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:35:25.153237    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae1e87d_459a_4914_8940_3afe662ac46c.slice/crio-a3b0da2b7135a7e37b735abc5360473c1f66b7b981174fbb2b35f70c1a5f3e05 WatchSource:0}: Error finding container a3b0da2b7135a7e37b735abc5360473c1f66b7b981174fbb2b35f70c1a5f3e05: Status 404 returned error can't find the container with id a3b0da2b7135a7e37b735abc5360473c1f66b7b981174fbb2b35f70c1a5f3e05
Nov 01 23:35:25.480743 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:25.480704    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b" event={"ID":"0ae1e87d-459a-4914-8940-3afe662ac46c","Type":"ContainerStarted","Data":"164bddb85ffdf8f68c6912a600019da435664d0d3023240270135762012cfa2b"}
Nov 01 23:35:25.481147 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:25.480795    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b" event={"ID":"0ae1e87d-459a-4914-8940-3afe662ac46c","Type":"ContainerStarted","Data":"a3b0da2b7135a7e37b735abc5360473c1f66b7b981174fbb2b35f70c1a5f3e05"}
Nov 01 23:35:25.585491 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:25.585446    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b" podStartSLOduration=1.585433766 podStartE2EDuration="1.585433766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:35:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:35:25.585412335 +0000 UTC m=+4306.903062335" watchObservedRunningTime="2026-11-01 23:35:25.585433766 +0000 UTC m=+4306.903083766"
Nov 01 23:35:26.488548 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:26.488513    4531 generic.go:356] "Generic (PLEG): container finished" podID="0ae1e87d-459a-4914-8940-3afe662ac46c" containerID="164bddb85ffdf8f68c6912a600019da435664d0d3023240270135762012cfa2b" exitCode=0
Nov 01 23:35:26.488983 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:26.488600    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b" event={"ID":"0ae1e87d-459a-4914-8940-3afe662ac46c","Type":"ContainerDied","Data":"164bddb85ffdf8f68c6912a600019da435664d0d3023240270135762012cfa2b"}
Nov 01 23:35:27.624963 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.624931    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:27.687500 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.687475    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca\") pod \"0ae1e87d-459a-4914-8940-3afe662ac46c\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") "
Nov 01 23:35:27.687645 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.687559    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca\") pod \"0ae1e87d-459a-4914-8940-3afe662ac46c\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") "
Nov 01 23:35:27.687645 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.687607    4531 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst\" (UniqueName: \"kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst\") pod \"0ae1e87d-459a-4914-8940-3afe662ac46c\" (UID: \"0ae1e87d-459a-4914-8940-3afe662ac46c\") "
Nov 01 23:35:27.688002 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.687970    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca" pod "0ae1e87d-459a-4914-8940-3afe662ac46c" (UID: "0ae1e87d-459a-4914-8940-3afe662ac46c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 23:35:27.689506 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.689475    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca" pod "0ae1e87d-459a-4914-8940-3afe662ac46c" (UID: "0ae1e87d-459a-4914-8940-3afe662ac46c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 23:35:27.689612 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.689594    4531 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst" pod "0ae1e87d-459a-4914-8940-3afe662ac46c" (UID: "0ae1e87d-459a-4914-8940-3afe662ac46c"). InnerVolumeSpecName "kube-api-access-h6tst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 23:35:27.789325 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.789247    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-xks-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:35:27.789325 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.789273    4531 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae1e87d-459a-4914-8940-3afe662ac46c-maas-api-openshift-ca\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:35:27.789325 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:27.789282    4531 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6tst\" (UniqueName: \"kubernetes.io/projected/0ae1e87d-459a-4914-8940-3afe662ac46c-kube-api-access-h6tst\") on node \"ip-10-0-130-144.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 23:35:28.501712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:28.501676    4531 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b"
Nov 01 23:35:28.501712 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:28.501702    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-rtp8b" event={"ID":"0ae1e87d-459a-4914-8940-3afe662ac46c","Type":"ContainerDied","Data":"a3b0da2b7135a7e37b735abc5360473c1f66b7b981174fbb2b35f70c1a5f3e05"}
Nov 01 23:35:28.501955 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:28.501769    4531 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3b0da2b7135a7e37b735abc5360473c1f66b7b981174fbb2b35f70c1a5f3e05"
Nov 01 23:35:36.530023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:36.529989    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f75d86897-txbsw_02c23e68-7466-4755-9a16-47f14ec6e7cd/authorino/0.log"
Nov 01 23:35:42.526456 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:35:42.526417    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:35:47.371694 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:47.371645    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-znqzz_f1d29b80-c299-450d-9e81-8bca05696da9/manager/0.log"
Nov 01 23:35:47.644739 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:47.644656    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qbc8z_dd9d1da5-3391-4195-b1b4-4a839c7232e6/manager/0.log"
Nov 01 23:35:48.868988 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:48.868959    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cl6j4_a515ca03-c82e-4b92-9103-57606ab3d027/proxy/0.log"
Nov 01 23:35:50.120669 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:50.120639    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rsmxq_f316c8b5-c1ff-4119-ae79-cf194fa662b0/revoke-keys/0.log"
Nov 01 23:35:50.346370 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:50.346344    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-rtp8b_0ae1e87d-459a-4914-8940-3afe662ac46c/revoke-keys/0.log"
Nov 01 23:35:50.566716 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:50.566695    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s65jg_f74f347d-671f-4320-9488-b5a92363d39d/revoke-keys/0.log"
Nov 01 23:35:50.778684 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:50.778659    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-glt96_734d44d5-7f85-4dee-a5ee-5edae6fbfd54/postgres/0.log"
Nov 01 23:35:52.103799 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:52.103768    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:35:52.213878 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:52.213853    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f75d86897-txbsw_02c23e68-7466-4755-9a16-47f14ec6e7cd/authorino/0.log"
Nov 01 23:35:52.627151 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:52.627122    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptxkz_4a74e855-f374-4e3a-ac80-233426d448fa/manager/0.log"
Nov 01 23:35:52.855673 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:52.855648    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-rjmf9_d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e/manager/0.log"
Nov 01 23:35:53.302082 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:53.302050    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7xs5r_07bfafb5-ea72-446c-8a1f-b8ea8172020d/manager/0.log"
Nov 01 23:35:53.850765 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:53.850737    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tqmt8_f8dc0e0e-3872-43b3-b0c5-69993fb224d9/manager/0.log"
Nov 01 23:35:54.621687 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:54.621660    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g_fe145b4b-7752-4d48-806b-424e365ad4af/istio-proxy/0.log"
Nov 01 23:35:55.751347 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:55.751316    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-4txmj_31480339-9503-4930-bfb6-3d8c4b2680ca/istio-proxy/0.log"
Nov 01 23:35:56.482179 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:56.482151    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-glttk_da2a2592-ce65-4dcc-add0-1031bc2a1079/istio-proxy/0.log"
Nov 01 23:35:56.977788 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:56.977752    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mdd8f_d344e962-a243-4156-b9aa-353322a53d47/istio-proxy/0.log"
Nov 01 23:35:57.853128 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:35:57.853102    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6_d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85/istio-proxy/0.log"
Nov 01 23:36:01.814821 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:01.814792    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f75d86897-txbsw_02c23e68-7466-4755-9a16-47f14ec6e7cd/authorino/0.log"
Nov 01 23:36:11.213394 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:11.213360    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-znqzz_f1d29b80-c299-450d-9e81-8bca05696da9/manager/0.log"
Nov 01 23:36:11.457144 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:11.457120    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-qbc8z_dd9d1da5-3391-4195-b1b4-4a839c7232e6/manager/0.log"
Nov 01 23:36:12.412540 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:12.412508    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-cl6j4_a515ca03-c82e-4b92-9103-57606ab3d027/proxy/0.log"
Nov 01 23:36:12.524829 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:36:12.524794    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:36:13.474663 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:13.474626    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-rsmxq_f316c8b5-c1ff-4119-ae79-cf194fa662b0/revoke-keys/0.log"
Nov 01 23:36:13.696699 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:13.696666    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-rtp8b_0ae1e87d-459a-4914-8940-3afe662ac46c/revoke-keys/0.log"
Nov 01 23:36:13.913070 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:13.912981    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-s65jg_f74f347d-671f-4320-9488-b5a92363d39d/revoke-keys/0.log"
Nov 01 23:36:14.128891 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:14.128847    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-glt96_734d44d5-7f85-4dee-a5ee-5edae6fbfd54/postgres/0.log"
Nov 01 23:36:15.208678 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:15.208652    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f75d86897-txbsw_02c23e68-7466-4755-9a16-47f14ec6e7cd/authorino/0.log"
Nov 01 23:36:15.620842 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:15.620771    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptxkz_4a74e855-f374-4e3a-ac80-233426d448fa/manager/0.log"
Nov 01 23:36:15.852371 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:15.852344    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-rjmf9_d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e/manager/0.log"
Nov 01 23:36:16.306323 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:16.306298    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7xs5r_07bfafb5-ea72-446c-8a1f-b8ea8172020d/manager/0.log"
Nov 01 23:36:16.794357 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:16.794335    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tqmt8_f8dc0e0e-3872-43b3-b0c5-69993fb224d9/manager/0.log"
Nov 01 23:36:17.366900 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:17.366873    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g_fe145b4b-7752-4d48-806b-424e365ad4af/istio-proxy/0.log"
Nov 01 23:36:18.542027 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:18.541997    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-4txmj_31480339-9503-4930-bfb6-3d8c4b2680ca/istio-proxy/0.log"
Nov 01 23:36:19.265840 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:19.265810    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-glttk_da2a2592-ce65-4dcc-add0-1031bc2a1079/istio-proxy/0.log"
Nov 01 23:36:19.733848 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:19.733825    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-mdd8f_d344e962-a243-4156-b9aa-353322a53d47/istio-proxy/0.log"
Nov 01 23:36:20.607992 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:20.607968    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7cc4b68895-n8mp6_d45d6b2e-944f-49a7-be3d-4b6dd7a0dd85/istio-proxy/0.log"
Nov 01 23:36:42.525532 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:36:42.525486    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:36:42.829020 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:42.828950    4531 scope.go:122] "RemoveContainer" containerID="b231eaf1531f184a16a7e69b4730830b83a0fa8970bb84c6d1b224bcae3ee583"
Nov 01 23:36:52.458468 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:36:52.458429    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:37:12.525579 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:37:12.525538    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:37:27.834505 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.834466    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wg4hr/must-gather-2xj7r"]
Nov 01 23:37:27.835159 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.835137    4531 cpu_manager.go:414] "containerMap: removing container" podUID="0ae1e87d-459a-4914-8940-3afe662ac46c" containerName="revoke-keys"
Nov 01 23:37:27.835251 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.835161    4531 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae1e87d-459a-4914-8940-3afe662ac46c" containerName="revoke-keys"
Nov 01 23:37:27.835331 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.835317    4531 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae1e87d-459a-4914-8940-3afe662ac46c" containerName="revoke-keys"
Nov 01 23:37:27.839271 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.839250    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:27.850146 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.850125    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca432930-7fb5-4db5-930d-bbff48fac6cc-must-gather-output\") pod \"must-gather-2xj7r\" (UID: \"ca432930-7fb5-4db5-930d-bbff48fac6cc\") " pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:27.850261 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.850162    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9vdq\" (UniqueName: \"kubernetes.io/projected/ca432930-7fb5-4db5-930d-bbff48fac6cc-kube-api-access-d9vdq\") pod \"must-gather-2xj7r\" (UID: \"ca432930-7fb5-4db5-930d-bbff48fac6cc\") " pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:27.873656 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.873632    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:37:27.888050 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.888032    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wg4hr\"/\"openshift-service-ca.crt\""
Nov 01 23:37:27.934545 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.934520    4531 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wg4hr\"/\"kube-root-ca.crt\""
Nov 01 23:37:27.934700 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.934680    4531 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wg4hr\"/\"default-dockercfg-76nmg\""
Nov 01 23:37:27.937486 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.937467    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/must-gather-2xj7r"]
Nov 01 23:37:27.951814 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.951791    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca432930-7fb5-4db5-930d-bbff48fac6cc-must-gather-output\") pod \"must-gather-2xj7r\" (UID: \"ca432930-7fb5-4db5-930d-bbff48fac6cc\") " pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:27.951919 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.951841    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9vdq\" (UniqueName: \"kubernetes.io/projected/ca432930-7fb5-4db5-930d-bbff48fac6cc-kube-api-access-d9vdq\") pod \"must-gather-2xj7r\" (UID: \"ca432930-7fb5-4db5-930d-bbff48fac6cc\") " pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:27.952102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:27.952085    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ca432930-7fb5-4db5-930d-bbff48fac6cc-must-gather-output\") pod \"must-gather-2xj7r\" (UID: \"ca432930-7fb5-4db5-930d-bbff48fac6cc\") " pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:28.005102 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:28.005082    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9vdq\" (UniqueName: \"kubernetes.io/projected/ca432930-7fb5-4db5-930d-bbff48fac6cc-kube-api-access-d9vdq\") pod \"must-gather-2xj7r\" (UID: \"ca432930-7fb5-4db5-930d-bbff48fac6cc\") " pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:28.090130 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:28.090075    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:37:28.152723 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:28.152699    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/must-gather-2xj7r"
Nov 01 23:37:28.523908 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:28.523862    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/must-gather-2xj7r"]
Nov 01 23:37:28.528310 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:37:28.528282    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca432930_7fb5_4db5_930d_bbff48fac6cc.slice/crio-c686248143c4362ad8e04ea460b1a1f3b6be60b3f81b07b891c6318601c3917c WatchSource:0}: Error finding container c686248143c4362ad8e04ea460b1a1f3b6be60b3f81b07b891c6318601c3917c: Status 404 returned error can't find the container with id c686248143c4362ad8e04ea460b1a1f3b6be60b3f81b07b891c6318601c3917c
Nov 01 23:37:28.530008 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:28.529990    4531 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 23:37:29.216244 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:29.216203    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/must-gather-2xj7r" event={"ID":"ca432930-7fb5-4db5-930d-bbff48fac6cc","Type":"ContainerStarted","Data":"c686248143c4362ad8e04ea460b1a1f3b6be60b3f81b07b891c6318601c3917c"}
Nov 01 23:37:30.223929 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:30.223894    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/must-gather-2xj7r" event={"ID":"ca432930-7fb5-4db5-930d-bbff48fac6cc","Type":"ContainerStarted","Data":"3594bc60efa1d6d4c3f3a58b25c3e759540c5cb178dfb4304456120b0a926ea9"}
Nov 01 23:37:30.224326 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:30.223945    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/must-gather-2xj7r" event={"ID":"ca432930-7fb5-4db5-930d-bbff48fac6cc","Type":"ContainerStarted","Data":"7ff5d6b52f86a3e43a4e736861774bdd737c4b13f8197d04c3bc6b5d01c75439"}
Nov 01 23:37:30.331632 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:30.331576    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wg4hr/must-gather-2xj7r" podStartSLOduration=1.9918406709999998 podStartE2EDuration="3.331562846s" totalImagesPullingTime="1.339722175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 23:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:37:30.329115629 +0000 UTC m=+4431.646765630" watchObservedRunningTime="2026-11-01 23:37:30.331562846 +0000 UTC m=+4431.649212848"
Nov 01 23:37:37.519759 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:37.518777    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v4lzz_ec65095d-39d7-47d4-8951-b9676b711b1d/global-pull-secret-syncer/0.log"
Nov 01 23:37:39.381992 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:39.381960    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log"
Nov 01 23:37:41.186023 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:41.185958    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-144.us-west-2.compute.internal_6ad307b3c1f78fefabe49820df30682a/haproxy/0.log"
Nov 01 23:37:42.535748 ip-10-0-130-144 kubenswrapper[4531]: E1101 23:37:42.535223    4531 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5qrr5" containerName="extractor"
Nov 01 23:37:42.851747 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:42.851641    4531 scope.go:122] "RemoveContainer" containerID="f13ece793b02cd52a2b59fe041d4c229cc2e45684e16145501380461e5c99570"
Nov 01 23:37:46.172320 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:46.172287    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f75d86897-txbsw_02c23e68-7466-4755-9a16-47f14ec6e7cd/authorino/0.log"
Nov 01 23:37:46.496104 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:46.496071    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ptxkz_4a74e855-f374-4e3a-ac80-233426d448fa/manager/0.log"
Nov 01 23:37:46.914524 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:46.914460    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-rjmf9_d5f01ba5-cb85-4957-b96b-d46eb9ef0d4e/manager/0.log"
Nov 01 23:37:46.942214 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:46.942178    4531 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"]
Nov 01 23:37:46.949982 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:46.949961    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.032766 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.032715    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"]
Nov 01 23:37:47.048181 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.048148    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xd45\" (UniqueName: \"kubernetes.io/projected/9cbc9758-f221-47ff-a673-05e45733e6d4-kube-api-access-9xd45\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.048335 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.048221    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-sys\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.048397 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.048371    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-proc\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.048436 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.048422    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-lib-modules\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.048552 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.048530    4531 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-podres\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.149447 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149409    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xd45\" (UniqueName: \"kubernetes.io/projected/9cbc9758-f221-47ff-a673-05e45733e6d4-kube-api-access-9xd45\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.149652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149462    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-sys\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.149652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149542    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-proc\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.149652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149579    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-lib-modules\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.149652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149625    4531 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-podres\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.149652 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149633    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-sys\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.150007 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149642    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-proc\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.150007 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149694    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-lib-modules\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.150007 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.149711    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9cbc9758-f221-47ff-a673-05e45733e6d4-podres\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.206778 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.206750    4531 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xd45\" (UniqueName: \"kubernetes.io/projected/9cbc9758-f221-47ff-a673-05e45733e6d4-kube-api-access-9xd45\") pod \"perf-node-gather-daemonset-rtxpr\" (UID: \"9cbc9758-f221-47ff-a673-05e45733e6d4\") " pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.263907 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.263875    4531 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:47.629245 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.629167    4531 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"]
Nov 01 23:37:47.634542 ip-10-0-130-144 kubenswrapper[4531]: W1101 23:37:47.634511    4531 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9cbc9758_f221_47ff_a673_05e45733e6d4.slice/crio-518490fd2d5a96d7234368073c6fd9f707eb5b8c3b953881e63bd191194cd964 WatchSource:0}: Error finding container 518490fd2d5a96d7234368073c6fd9f707eb5b8c3b953881e63bd191194cd964: Status 404 returned error can't find the container with id 518490fd2d5a96d7234368073c6fd9f707eb5b8c3b953881e63bd191194cd964
Nov 01 23:37:47.775254 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:47.775229    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-7xs5r_07bfafb5-ea72-446c-8a1f-b8ea8172020d/manager/0.log"
Nov 01 23:37:48.430906 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:48.430874    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr" event={"ID":"9cbc9758-f221-47ff-a673-05e45733e6d4","Type":"ContainerStarted","Data":"d68271f31733ff89cde8e4ebfe8ea2f42b654121819b2a012368b7408d666980"}
Nov 01 23:37:48.431374 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:48.430936    4531 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr" event={"ID":"9cbc9758-f221-47ff-a673-05e45733e6d4","Type":"ContainerStarted","Data":"518490fd2d5a96d7234368073c6fd9f707eb5b8c3b953881e63bd191194cd964"}
Nov 01 23:37:48.431374 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:48.430981    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:48.538668 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:48.538616    4531 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr" podStartSLOduration=2.538601652 podStartE2EDuration="2.538601652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 23:37:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 23:37:48.536705935 +0000 UTC m=+4449.854355937" watchObservedRunningTime="2026-11-01 23:37:48.538601652 +0000 UTC m=+4449.856251653"
Nov 01 23:37:48.640006 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:48.639977    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-5778bb4449-tqmt8_f8dc0e0e-3872-43b3-b0c5-69993fb224d9/manager/0.log"
Nov 01 23:37:51.254494 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:51.254422    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/alertmanager/0.log"
Nov 01 23:37:51.671568 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:51.671506    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/config-reloader/0.log"
Nov 01 23:37:52.088700 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:52.088671    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/kube-rbac-proxy-web/0.log"
Nov 01 23:37:52.506061 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:52.506028    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/kube-rbac-proxy/0.log"
Nov 01 23:37:52.732237 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:52.732199    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:37:52.927928 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:52.927853    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/kube-rbac-proxy-metric/0.log"
Nov 01 23:37:53.345322 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:53.345297    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/prom-label-proxy/0.log"
Nov 01 23:37:53.762901 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:53.762878    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c3075de8-ae16-4b2d-9199-d31c4726025f/init-config-reloader/0.log"
Nov 01 23:37:54.450760 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:54.450715    4531 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wg4hr/perf-node-gather-daemonset-rtxpr"
Nov 01 23:37:54.607059 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:54.607030    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7zwd6_509dedcc-8556-414c-b033-c1bb924bec68/kube-state-metrics/0.log"
Nov 01 23:37:55.020035 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:55.020003    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7zwd6_509dedcc-8556-414c-b033-c1bb924bec68/kube-rbac-proxy-main/0.log"
Nov 01 23:37:55.437817 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:55.437704    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7zwd6_509dedcc-8556-414c-b033-c1bb924bec68/kube-rbac-proxy-self/0.log"
Nov 01 23:37:57.925212 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:57.925180    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n97zh_8d78fe3b-1aae-4121-9b0d-36d1eb1ece95/node-exporter/0.log"
Nov 01 23:37:58.342852 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:58.342813    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n97zh_8d78fe3b-1aae-4121-9b0d-36d1eb1ece95/kube-rbac-proxy/0.log"
Nov 01 23:37:58.758678 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:37:58.758647    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-n97zh_8d78fe3b-1aae-4121-9b0d-36d1eb1ece95/init-textfile/0.log"
Nov 01 23:38:00.441495 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:00.441464    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cjzrk_11dc5494-1fd6-43c6-89d7-80fdd7b284bd/kube-rbac-proxy-main/0.log"
Nov 01 23:38:00.857202 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:00.857173    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cjzrk_11dc5494-1fd6-43c6-89d7-80fdd7b284bd/kube-rbac-proxy-self/0.log"
Nov 01 23:38:01.273110 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:01.273085    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-cjzrk_11dc5494-1fd6-43c6-89d7-80fdd7b284bd/openshift-state-metrics/0.log"
Nov 01 23:38:05.384079 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:05.384049    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-cv4xj_4db06e67-53f1-49f8-8089-a13799623055/prometheus-operator-admission-webhook/0.log"
Nov 01 23:38:05.807313 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:05.807281    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-566c444b98-tzc48_b3945423-2879-49c0-bee4-03f8e0902a9e/telemeter-client/0.log"
Nov 01 23:38:06.225262 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:06.225208    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-566c444b98-tzc48_b3945423-2879-49c0-bee4-03f8e0902a9e/reload/0.log"
Nov 01 23:38:06.644109 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:06.644036    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-566c444b98-tzc48_b3945423-2879-49c0-bee4-03f8e0902a9e/kube-rbac-proxy/0.log"
Nov 01 23:38:07.066353 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:07.066325    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5c7bdb56-zp6cq_030ddc17-a369-45c0-9dc0-09c563cb22cc/thanos-query/0.log"
Nov 01 23:38:07.481496 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:07.481468    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5c7bdb56-zp6cq_030ddc17-a369-45c0-9dc0-09c563cb22cc/kube-rbac-proxy-web/0.log"
Nov 01 23:38:07.900277 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:07.900207    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5c7bdb56-zp6cq_030ddc17-a369-45c0-9dc0-09c563cb22cc/kube-rbac-proxy/0.log"
Nov 01 23:38:08.337314 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:08.337283    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5c7bdb56-zp6cq_030ddc17-a369-45c0-9dc0-09c563cb22cc/prom-label-proxy/0.log"
Nov 01 23:38:08.668025 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:08.667958    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6fd5fcc55chbg7g_fe145b4b-7752-4d48-806b-424e365ad4af/istio-proxy/0.log"
Nov 01 23:38:08.762143 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:08.762101    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5c7bdb56-zp6cq_030ddc17-a369-45c0-9dc0-09c563cb22cc/kube-rbac-proxy-rules/0.log"
Nov 01 23:38:08.868815 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:08.868787    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:38:09.057475 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:09.057440    4531 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-ggq5x_1c493649-a226-4fdc-a95a-ba1a9de15539/konnectivity-agent/0.log" limit=1024
Nov 01 23:38:09.242810 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:09.242788    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f5c7bdb56-zp6cq_030ddc17-a369-45c0-9dc0-09c563cb22cc/kube-rbac-proxy-metrics/0.log"
Nov 01 23:38:10.508855 ip-10-0-130-144 kubenswrapper[4531]: I1101 23:38:10.508822    4531 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7f694df8bc-4txmj_31480339-9503-4930-bfb6-3d8c4b2680ca/istio-proxy/0.log"